I don't think it's a bad way to benchmark new models, I just find it concerning that the author implies that "pelican on a bicycle" has been exhausted.
At the risk of making overly broad, unfalsifiable claims I think multi-year exposure to AI content has dramatically raised our expectations for speed and volume but lowered them for quality.
We see a very janky pelican and declare the problem solved.
I don't think he's claiming it's been exhausted. It's just that things have progressed to a point where people are arguing over the finer points of which pelican looks better -- which is often a matter of taste, and an indication that we've hit the knee in benchmark where models are no longer failing in obviously awful ways.
I haven't really seen evidence that any ai can reliably draw a pelican riding a bicycle. Not if you look at the image long enough to take it in. Even the best ones have something wrong with them. Not a matter of taste but a matter of having both legs peddling on the viewer's side of the bicycle or having two beaks.
I'm actually beginning to wonder if some people who ignore these things have a different, somewhat lesser ability to percieve image details than I do.
I mean I guess its fine to go on to another test despite never actually passing the pelican bike test, but there's a sense that we have to use another test because AI is now good at pelicans on bikes, which is just not true.
AI has deeply changed the way I think, feel and act around a computer. In the same way that dialing into the internet changed things for me. Since using ChatGPT the first time until now I have never cared once to look at these pelicans on bikes people seem to get hung up about. It could never have been a thing and nothing would change. See the forest through the trees.
What you’re saying is that you’re not interested in benchmarks. But then you go a step further and state that this particular benchmark is entirely inconsequential. That’s like telling you that if you didn’t exist, nothing would change. Even if that were true, it would still be an insensitive and rude thing to say, wouldn’t it?
Usually when I say insensitive things there are more downvotes then upvotes. That isn't the case here. I might be rubbing against a truth somewhere here.
When it started, it was clear what LLM would stand out, its style, etc.
Nowadays, the pelicans look similar, the difference is in details and sometimes hard to catch. Sure, the task is not completed perfectly, but that's not the point. It was supposed to be a benchmark to quickly benchmark a LLM against others.
Many humans would struggle with this even with very good tooling (ie not writing raw svg and using illustrator). I struggle to draw a bicycle accurately. But yes, I suspect it will be diminishing returns and I doubt it will ever be perfect due to the average nature of AI but I’d like to be wrong.
> Many humans would struggle with this even with very good tooling
But no ones hire random humans for things like this. You go and hire a vector artist and they will get your a very good pelican on a a bike. That's how you get things done when you can't do it.
So what if it is more profitable or valuable? It is still not better. Something being more profitable/valuable does not make it better, just like something being better does not make it more profitable/valuable. Sometimes, in some pursuits, for some outputs under some circumstances, the two are correlated. In others, the two are anti-correlated.
>If the computer can't do it better than a human being, then what's the point?
Because the benchmark wasn't testing "can an LLM draw a pelican like a human". The original article was testing the relative capabilities between LLMs. Now that LLMs can all draw pelicans all similarly, the test is less interesting as a comparative benchmark.
I don't know how much money has been spent for AI, and I very much doubt you do. Do you know if more has been spent on LLM the last 9 years -- since "Attention is all you need" -- than Internet infrastructure during, say 1995 to 2004? That included the dotcom crash. Did you lament how a failure the Internet was?
LLM has progressed a lot in the last two year, judging from the pelican drawings. I personally couldn't care less about it though. I do know that I've gone from using no AI at all for coding to probably 95%. I hardly code by hand anymore. That's much more impressive and significant. Failure you said?
It’s better than many of the AI offerings and the bike could steer and the ducks are sitting on saddles, but the three nephews can’t reach the bottom of the pedal stroke and by the looks of their feet on the far side of the bike they aren’t trying. That shouldn’t satisfy Donald and Daisy, leaving them with all the work.
I think it's touching the limit of what one can reasonably expect any intelligent thing to produce with the only direction being "produce an svg of a pelican riding a bicycle".
When you aren't sure if an LLM can write an svg well, or that it will be able to form a pelican shape, or animate a bicycle, it's a good test. After that, it's all judgement: how detailed should the pelican be? pelicans are the wrong shape for a bicycle by default, so how much can I change its physiology to match using a bicycle before it isn't a pelican? Do I care about how well the client is able to render complex geometry?
It's not that there isn't room to do better, or that it doesn't tell you anything at all, but rather we've reached a point where what it tells us isn't very clear anymore.
> I think it's touching the limit of what one can reasonably expect any intelligent thing to produce with the only direction being "produce an svg of a pelican riding a bicycle".
Oh come now. I am extremely confident that if I hired a professional artist to draw a picture of a pelican riding a bicycle, I would get something inarguably much better than what today's best coding LLMs can produce.
At that rate you could use an image model, which was designed for the task. Thats the absurdity of this test. It’s often a text only generation model that has never seen a pelican, coerced into creating an xml graphical representation that a human might recognize. That it does anything passable is already astounding.
It probably has a few million inputs on how a pelican and bicycle looks, not to mention the amount of data on how to create SVG’s. Ask it to create a relaxing spa website and it will, even though it has never seen a spa.
You’re right. Modern frontier models are now multimodal. I used often as weak a hedge, because I know at least his gpt3.5 turbo and llama3.1 generated pelicans were from text only models without image training. The chinese models are interesting, because before their vision models existed they may have been distilling text only models from text output of American vision models, so they could have benefited from the teacher model’s vision capability without being vision models themselves.
> if I hired a professional artist to draw a picture of a pelican riding a bicycle,
I think AI folks have done a terrible job of communicating this, but replacing a professional simply isn't the point. The point is to serve all the situations where people would've never considered hiring a professional, and where perfection or artistic merit isn't the point (say a personal throwaway recreation of an LOTR world).
And I think in that regard the benchmarks are pretty good.
Could you share any examples that come close to those limits? I haven’t seen any that don’t have obvious flaws in proportions, layering, composition, color palette, visual clutter, or stylistic consistency.
> touching the limit of what one can reasonably expect
If the expectation is that AI is going to replace "knowledge workers" then the limit would be a darn perfect drawing. We are nowhere close to that.
And Elon is already propagating the age of abundance where money won't exist anymore, right before calling the interviewing journalist dishonest and deservedly losing public trust. Smh my head.
> If the expectation is that AI is going to replace "knowledge workers" then the limit would be a darn perfect drawing. We are nowhere close to that.
What knowledge workers do you know that have excellent drawing skills? I worked in a design agency and for a couple of years, each week me and a few other people would attempt to sketch a member of our group: one person would be the model and sit still, and everyone else would draw her/him.
Let me tell you, if producing a convincing portrait was a prerequisite for being a knowledge worker, there would be 99% fewer knowledge workers.
It is a proxy for intelligence. And as long as that metric is not gamed (which it surprisingly doesn't appear to be yet), the drawing skill of sth is a quite reasonable test.
It surprises me how many people in this community don't get this. Obviously, most prompts thrown into an AI chatbot/interface are about something no knowledge worker would ever have to deal with. That doesn't disqualify them as a tool for measuring progress of the models.
Also, one of the advantages of the pelican test is that you can evaluate it all at once. There’s no reason the two-dimensional depiction can’t be made more challenging. Yes, at some point the pelicans might approach the subjectivity of a fine art painting, but we haven’t even seen a depiction that’s competent by the standards of a high school art class. That’s not to say the elementary school–level SVGs aren’t amazing - rather, I agree with your point.
The bike is generally okay, apart from medium which derped hard. Max has correct diamond, correct head tube, and so on. Only nitpicks are that the front fork offset isn't there and the chain doesn't touch the rear sprocket correctly.
Can someone explain what the pelican on a bicycle tests exactly? And why is it so important? I've never understood how it could translate to a useful task in real life.
On the most basic level, its asking the ai to generate valid svg code for a picture of a pelican riding a bicycle, as a way of checking its intelligence. Popularized (invented?) by simonw, its been used as part of his reviews of new models as they come out since oct 2025.
It used to be a very difficult task for models, see [2,3,4]
it cuts across several tasks that AI used to be very bad at, but now has improved quite a bit. Namely, spatial reasoning (because it has to manually place the points of the svg such that they make sense and form what it says it forms. This used to not work very well, with random shapes floating around that it would mark things like "eyebrows" but were nowhere near the "eyes", etc.
It also tests the model's world knowledge (what do pelicans look like? sure they have wings, feet, beaks etc, but what shape are they? how to get proportions roughly right? this isn't a given from text data about the bird. This goes doubly for a bike, which is a quite complex shape that most humans fail to draw correctly[1] (many draw the frame or chain connecting in impossible ways that would not ever function mechanically)
Before it was pelican on a bicycle there were people having it do horses/unicorns making the rounds - gpt4.0 or whatever would often make hideous abominations of legs and mouths
If your school mascot is a pelican and you need to make a flyer for the upcoming bike safety event, then this precise thing becomes useful. Most things that are useful in the real world seem not useful out of context.
Humans are drawing pelicans riding bicycles now. Just google it and you will find 5 or 10 of them in the first few results. Including a t-shirt design.
What does this even test for? Can I use LLMs to directly generate machine code to replace my compiler? Or maybe I can use LLMs as a bare metal OS / scheduler to replace my machine's operating system and scheduler? It makes zero sense to test for that.
Not only that this so-called "benchmark" isn't economically useful, but that it tests for the sake of testing; and for attention.
To end this obsession with generating SVG pelicans, Quiver AI's [0] model is actually designed to generate SVGs from prompts and has done so for years.
So there is no need to continue with this un-serious pseudoscientific "benchmark".
So if we test models that only output text to directly generate waveforms of sound from text or binary code, or directly generating binary code to replace a compiler, does that mean it is "intelligent"?
Does that even test for intelligence?
This is like testing if a horse can fly just because someone showed an image of a Pegasus, or testing if a fish can climb up a tree and believing they are not intelligent because each of them cannot fly or climb up trees.
I'm not sure where you are coming from, but we are testing a model that can create images to create an image for us. If it can't even do that well then I'm not sure why we need to talk about horses here.
Pulling things into the ridiculous isn't condusive to a good faith discussion and does not help proving pseudoscientificness which you seem to be after. (I don't belive that the pelican bicycle test is meant to be a serious scientific endeavour, btw.)
> I'm not sure where you are coming from, but we are testing a model that can create images to create an image for us.
Then use a model that supports multi-modal output then, instead of using models that only support raw text output to construct an image.
> If it can't even do that well then I'm not sure why we need to talk about horses here.
Testing a model that only outputs text and forcing it to output an unsupported modality is obviously not going to work well which is what it going on here.
Would you expect Claude/GPT to replace your compiler because it can output text? That would be like expecting that horses can fly because someone saw an ancient winged-horse on a brick tablet.
> Pulling things into the ridiculous isn't condusive to a good faith discussion and does not help proving pseudoscientificness which you seem to be after.
That is because the premise of the test is ridiculous and pseudoscientific.
> (I don't belive that the pelican bicycle test is meant to be a serious scientific endeavour, btw.)
Why did you say it was a "test for intelligence", when we both know it is a joke that tests for nothing?
One way to characterize intelligence is the capability of solving problems one has not specifically been trained for.
Somebody who has trained painting animals and vehicles for decades is surely skilled at that, but is not solving new problems, so is not necessarily applying intelligence when painting something of that sort.
You can take a reasonably skilled programmer, show them some C code and ask them to write equivalent ASM code, perhaps providing them with a language reference for either, and they will be able to do this translation. Slowly, but they will get there. They have not been trained to do this, but they will "figure it out". With enough time and motivation, even non-programmers would be able to do that. That is what intelligence is and of course Claude/GPT needs to be able to do that if it wants to claim that it intelligent. No flying horses.
