Tl;dr: The trouble with ntile seems to be that, when the author of the post imagined what it did based solely on its name and she imagined wrong, she ran with her imaginary version way too long before checking the documentation, which was very clear about its behavior:
> Unlike other ranking functions, ntile() ignores ties: it will create evenly sized buckets even if the same value of x ends up in different buckets.
I’m pretty sure NTILE is just a generally cursed function. SQL NTILE also requires everything to be loaded in memory because of the odd rule that larger buckets precede smaller buckets, so it’s unusable on anything decently sized.
Not being able to specify how ties are handled (and choosing, as far as I can tell, a fairly useless definition) fits the bill.
If it is "cursed" for a function to do what it is clearly documented to do, instead of what someone mistakenly imagines it to do, then what function isn't cursed?
> Not being able to specify how ties are handled (and choosing, as far as I can tell, a fairly useless definition) fits the bill.
Actually, you can completely specify how ties are handled—and should, in any study designed to be repeatable. The docs for dplyr::ntile tell us that:
> To rank by multiple columns at once, supply a data frame.
So, repeatable, completely specified tiebreaking is as easy as adding a tiebreaker column to the dataset, using whatever strategy makes sense for your study. For example, if we wanted random tiebreaking using R's built-in `runif`, all it takes is one extra line of code:
The explanation of "What's that extra 1 for?" in the column representation of 3-d coordinates (x y z 1) could benefit from mentioning that translation—moving things—is not a linear transformation (the origin is not mapped to itself) but an affine transformation. Therefore, you cannot represent translation in 3-d space with a 3x3 matrix. What you can do, though, is embed that 3-d space within a 4-d space fixed at some coordinate on its 4th dimension, typically w=1. Then, a translation in the original 3-d space can be represented as a linear transformation in the 4-d space and thus can also be represented by a 4x4 matrix multiplication. So the extra 1 is actually what allows all common 3-d operations, including translation, to be done via linear algebra and thereby harness the brutal power of matrix multiplication on modern computing devices.
You can do rotations and translations "via linear algebra" without homogeneous coordinates (the 4-element tuple representing a 3d point or vector), since adding two vectors is linear algebra.
What this lets you do is composing multiple transforms into one matrix multiplication instead of a sequence of multiplications and additions; that's what dramatically increases performance, on modern computing devices but most especially on ancient ones, where we were fighting for every MUL.
You don't need to use homogeneous coordinates to represent a series of linear transformations as a single matrix. That's just basic linear algebra: any series of linear transformations is itself a linear transformation, and any linear transformation can be represented as an equivalent transformation matrix.
But you do need homogenous coordinates if you want to include translations (fixed distance shifts, such as moving the camera) in the set of operations you can represent as linear transformations and thereby gain all the benefits of linear algebra, including the benefit of being able to include them in a series of operations that you can represent with a single transformation matrix.
Rick Szeliaki’s “Computer vision algorithms and applications” has a lot of great motivation for homogeneous coordinates in the first chapter and appendix. For example, in 2D, the cross product between two points in homogeneous coordinates is the line joining them, and the cross product between two lines in homogeneous coordinates is their point of intersection.
nice intuition there, this comment prompted me to consider a simpler example, 2d embedded within 3d. does the 2d plane (embedded in 3d) go through the 3d origin (where 0 maps to 0) and is thus a linear transformation in 3d but a 2d affine transform in 2d? it feels like this is the case?
No, the 2d x-y plane in your example cannot pass through the 3d space’s origin because that would imply that you fixed the z coordinate at zero. The plane must be fixed at some nonzero z because you need to be able move x and y values by some scaled version of z to make translation happen. If z is zero, that scheme does not work.
Consider a transformation f where we wish to move x-y coordinates s units to the right. In 2d, we could express it as:
f(x, y) = (x + s, y)
But that transformation is affine not linear. There is no way to generate the value s as a linear combination of the inputs x and y. So, our workaround is to embed the x-y plane into 3d space at z=1. Then we can move (x,y,1) points in that plane s units to the right using this transformation:
f(x, y, z) = (x + s*z, y, z)
This new transformation is linear: it maps (0,0,0) to itself. But it maps our embedded 2d plane's origin (0,0,1) to (s,0,1), shifting it right by s units, as we want.
The matrix form of that transformation is:
[[1 0 s]
[0 1 0]
[0 0 1]]
The same scheme would work if we had embedded the plane at any fixed z=r for nonzero r. We would only have to rescale the s in the matrix to s/r. Again, however, if r=0, this scheme will not work, as 1/r has gone to infinity.
ive been thinking -- if i have a 3x3 matrix [1 0 q; 0 1 r; 0 0 1], when dotted with x, those rows are planes in 3d with normals n1=(1 0 q), n2=(0 1 r) and n3=(0 0 1). plane 3 is parallel to the x-y plane and 1 unit up. plane 1 is tilted by q and parallel to the y-axis, plane 2 is tilted by r and parallel to the x-axis. the intersection of those planes (i.e. the solution x) when calculating Ax=b gives an output vector b sitting in 3d space at b=(x+qz, y+rz, 1*z). since we always specify z=1 we have b=(x+q, y+r, 1). in 3d this is a linear shear because we are translating proportionally by z but because z always equals 1 in this case we effectively get a translation in 2d.
so as the other 2 helpful commenters also just said: 3d shears using linear algebra degenerate to 2d affine transformations when z=1 (or w in 4d)
re-read this a day later and ofc it still makes sense to me but really most of it is barking up the wrong tree, focusing on solutions (which we already know) rather than outputs. if i could delete it i would but oh well.
Is that why quaternions are interesting for computer graphics? I studied mathematics and computer science but never understood (or even looked so hard into) the connection.
Kind of! The scalar allows vectors and scalars to be expressed together as one object, which has a lot of computational and mathematical niceties.
The killer feature is that you can put the two together for a rotation axis (~ vector) and angle (~ scalar).
With just three gimbals (rotating circles), if gimbals A and B are aligned, you only have two degrees of freedom (rotating A is the same as rotating B). Because of this, interpolating angles is unwieldy in vector space. 'Gimbal lock' confounds animation (in hilarious but unrealistic ways) but also aerospace (four hours before 'one small step for man', just after landing, Collins joked he would like a fourth gimbal for Christmas).
Another is Kalmia latifolia, the mountain laurel, the state flower of Pennsylvania. According to records at the Carnegie Museum of Natural History [1], "Benjamin Smith Barton (an American botanist in the late 1700s) wrote that in the autumn and winter of the year 1790, many people died in Pennsylvania from the effects of wild honey, collected from Kalmia plants."
The problem with alternative query languages is that the people who have the most knowledge about creating queries and of the relational domains underlying their businesses are all experts in SQL. Introducing something else, then, means your most natural user base must migrate away from something they understand how to use well, and that's a hard sell.
So, until the ultimate query language is developed, I'll take SQL with pipes. It's an easy sell and good enough to eliminate 90% of my gripes about SQL.
This is exactly the same problem facing people trying to develop new music notations. In order to grasp the domain enough, they have to be experts in the existing music notation, and once you're an expert in it, the motivation to create something new goes away. From what I've seen, the people who want a new music notation are mostly people uncomfortable with sight reading.
Experts have been criticizing SQL since it was a proprietary IBM language. Take this typewritten rant from 1983 [0] as an example. And we've had better query languages for just as long, e.g. datalog. People really love SQL though, which I can only assume is because the vast majority of usecases are slight variations on SELECT * FROM table.
For example, Elastic.
Much extra learning curve for little obvious gain.
I like to say, with zero research basis, that the New Shiny has to be an order of magnitude better than the Old Thing for people to say "Oh yeah, I gotta have that."
> Please please suggest me some books where language and style of speaking sentences takes the crown rather than the story being the selling point.
I will give you approximately twenty delightful novels where the prose and the stories are both excellent: Patrick O’Brian’s Aubrey–Maturin series.
I avoided these books for the longest time because I didn't believe I was the kind of person who would read “books about tall ships.” But so many people recommend these particular books because of the masterly writing that I tried the first, Master and Commander. Immediately, I was taken by O’Brian’s writing: vivid, lively, and engaging, while also utterly precise.
Give that first novel a try. It’s not easy, but it is oh so rewarding.
Edited to add some examples from the first novel:
“I have never yet known a man admit that he was either rich or asleep: perhaps the poor man and the wakeful man have some great moral advantage.”
“Where there was no equality there was no companionship: when a man was obliged to say ‘Yes, sir,’ his agreement was of no worth even if it happened to be true.”
“He held up two fingers, in case a landman might not fully comprehend so great a number.”
“But you know as well as I, patriotism is a word; and one that generally comes to mean either my country, right or wrong, which is infamous, or my country is always right, which is imbecile.”
I'm not who you asked, but I moved to Pittsburgh for a job, figuring I would earn money for a few years then move to real city. That was over thirty years ago. Turns out, I love the place.
Pittsburgh has most of the benefits of a larger city without most of the problems. Its infrastructure is in many ways over-provisioned, being built for past industry and a larger population (but the maintenance burden is catching up). Its unusually hilly and berivered geography has allowed its neighborhoods to escape homogenization and retain their individual ethnic characters. And the setting, with rivers, bridges, trails, forests, is beautiful. Plus the airport recently got overhauled to meet the region’s actual needs, making it way more convenient.
Oh, if you're in to sports, Pittsburgh has some great teams and a great fan base.
Speaking of over provisioned, you'll also find some very affordable housing there relative to the jobs available. So you could work at Google/Meta/Amazon/apple and have a 30 minute commute to work and pay $600k for a large house in a nice neighborhood on a nice sized plot of land, when the the best you could do is $2.2m for a less nice house in the bay area
Yeah, most people probably underestimate the effect that several things have on where they end up in life: education, self discipline, hard work, making good decisions, persistence, and, especially, chance. If you screw up majorly on any the factors under your control, you dramatically reduce your chances of “the good life.” But, even if you do all of them right, you are still at the mercy of chance. Get born in the wrong year or at the wrong place or into the wrong family or with the wrong genetics and that's all it takes to set you up for a hard life.
The tragedy is that we could have a society that gives people more opportunities to escape the downward pull of chance gone wrong early in life. Maybe someday we'll get there.
I really wanted to be a journalist when I was young, but then the PC era began and I became sort of addicted to computers and developed my first commercial website at a very young age.
I remember making more money than most people in my family (not a high bar, really) when I was working as a 20-year-old developer.
That led people in my family to think I was some sort of brilliant young man, but that bar was easily cleared (and still is) by pretty much anyone who had an interest in computers and applied themselves to pursue a career in the field.
My life would have been very different if I'd had a few more years to pursue journalism, which was already a very difficult field back then.
I wanted to be a professional artist, when I was younger[0]. I found out that even a mediocre programmer made more than even a relatively successful artist, and I enjoyed coding, so I stuck with that.
I also think that we should have mercy for people who’ve made mistakes. People who’ve willingly done dumb things, especially when they were young.
I feel this way because I did dumb things when I was young, and I’m sure you did too dear reader, and at least for me, I was lucky that those dumb things didn’t end in catastrophe.
It’s like it’s human nature to forget the dumb things you did if you were lucky enough to escape the consequences, and look down on those who weren’t so lucky.
> I also think that we should have mercy for people who’ve made mistakes.
Yes. When people make mistakes, they sometimes fall into holes that are hard to climb out of. It is in society’s interest to place ladders in these holes so that these unfortunate people can climb out and become productive members of society.
Instead, we have self-interested parties who have figured out how to extract rents from keeping people trapped in holes. Some of these parties are preventing ladders from being built. Others are destroying existing ladders. Others still are even inventing deeper, stickier holes—crypto scams, ubiquitous gambling, addictive gas-station drugs, dopamine-hit apps—to trap even greater numbers of their fellow human beings.
When a society allows the suffering of others to become a profitable business model, good things rarely follow.
This whole thing is one of those fruitless arguments about a badly defined notion of "most people".
For instance the scope of the "especially chance" assertion has strange effects. Does "get born in the wrong year" include 300,000 years of human history? In that case chance has very little effect. You're overwhelming likely to grow up using stone tools, with a 1 in 3 chance of also having access to pottery.
But that would be silly and obviously wasn't the intended question. The intended question is one that's closer to home, a question that we care about. But how much closer to which home, and what exactly is the scope of the question? It can be made less about luck by filtering out the unlucky from the scope.
Wrong year as in landing on a giant generational recession at the time period in life when people usually accelerate or maybe a world wide pandemic landing in your formative years.
Wrong year is a much smaller time scale. An example more close to home on this site, you/me just missed the free money cash days of the fango mango tech companies. Sure, there are still some of those jobs but far fewer with where interest rates are at.
I used to think a wrong year was maybe 1:80 but at 50, I've seen some unpleasant shit every decade.
I keep thinking about some odd-ball numerology weirdo I met on a trip in GG park about cycles of the universe and prime number harmony.
In the real world there are just boom/bust cycles, make hay while the sun shines. IME you'll be part of 8 cycles, 5 of which you can control a small part of.
For the vast majority of higher-level jobs you need certifications, not knowledge. There are very few jobs that you can get by simply demonstrating that you know something.
The reason people pay $100k for a year of college education most certainly isn’t the lectures, which you can often find on YouTube these days.
Most of life is luck, with regards to success and economic prosperity. We’ve built a system that ignores that for the benefit of a privileged few, using fairy tales around exceptionalism and faux meritocracy. We should build socioeconomic systems where luck is less a component for survival and living a good life, imho. I think we’re on our way there (support of democratic socialism and their candidates is at record highs, mostly around the young, and the old age out at a rate of 2M per year 55+ [higher support of capitalism because they were lucky under it]), but it’s going to take another decade or two to see the improvements (political cycles, electorate turnover, policy implementation lag, etc.)
Yes and no. I can think of easily three decision or inflection points in my life where if i went left instead of right, id be a multi millionaire. I can think of even more where my life direction would have been radically different.
Luck plays a huge part on where you start and land but no one is totally unlucky. Luck strikes everyone across all starting points.
What i have seen with my loved ones that struggle is that they have had many lucky breaks and somehow keep squandering them in a sad self fulfilling prophecy. Success begets success and failure begets failure. Both cycles are reversible.
Keep in mind “success” is not rich/famous but financially stable.
Luck is just as important even with perfect economic equality. It’s just that economic luck gets replaced by luck of genetics, luck of immaterial circumstances, etc.
I have always found the assumption strange that one is somehow preferable to the other, but most people seem to believe that, so maybe one day they’ll get what they’re asking for and then find out that it’s not what they wanted after all.
You must be joking. Democratic Socialism is just Socialism with a prettier name. Socialism does not work, full stop, unless you're in the group at the top controlling the strings. Don't point at the nordic countries, they have far less progressive taxation policies that the US when you factor VAT in.
I thought it was more like, if it doesn’t work it’s socialism, if it does work it’s not socialism? This is why Norway is at the same time called not socialism (because it works) and socialism (because conservatives don’t want us to even try adopting what works in Norway).
> Please, no. Don't blog about things you don't understand yet.
Do you really mean this? Or do you mean "When you blog, be honest. Don't pretend to understand things you don't."?
Blogging about things you are learning about, especially if you invite feedback, is a great way to learn and to build a shared understanding within a larger community.
But I agree 100% on the fake-it-till-you-make-it posts on blogs and LinkedIn. That stuff is a blight on the web.
"Honesty is a skill most people don't appreciate." This phrase starts an excerpt from a Piter Ralston interview that was one of turning points for me. I was caught by the word "skill"; not "virtue", as honesty normally called, but "skill", something from an entirely different area, as I thought. (https://www.youtube.com/watch?v=3TvZlpKT2ag).
I agree as long as you are very upfront about it being a learning/stream of conciousness blogpost, then it should be fine to watch how someone's learning journey progresses.
The author does outline the steps they take to make it clear
> Third, I try to make it clear on my blog who I am and what my credentials actually are. Even if it’s not explicitly described in the post, I have my real name and resume available on my /about page, so I don’t think a careful reader could be easily fooled into thinking I’m an expert on 19th-century England or space physics or LLM economics or anything like that.
If however that part is hidden and it is some confidently incorrect or pseudo guru type of post that is polluting the web with misinformation, then that is a different story.
Maintaining a high signal-to-noise ratio is important to some people, and they may completely disregard the musings of people outside their field of expertise.
> Unlike other ranking functions, ntile() ignores ties: it will create evenly sized buckets even if the same value of x ends up in different buckets.
reply