![\"A](\"https:\/\/falcoxai.com\/main\/wp-content\/uploads\/2026\/07\/are-ai-labs-pelicanmaxxing-th-inline-1.jpg\")
What the Pelicanmaxxing Benchmark Actually Measures<\/h2>\nThe origins of the pelican-on-a-bicycle test<\/h3>\n
Simon Willison\u2019s pelican-on-a-bicycle test began as a humorous way to gauge AI creativity, but it quickly became a viral benchmark. The test asks models to generate an SVG of a pelican riding a bicycle, a prompt that is simple in wording but complex in execution. It has since become a go-to example for evaluating AI outputs, even though it was never designed to measure real-world capabilities.<\/p>\n
The test gained traction because it\u2019s easy to score and hard to fail. Models can often generate a basic image with minimal effort, which makes the benchmark appealing for labs looking to show progress. However, this ease of scoring doesn\u2019t necessarily reflect the model\u2019s ability to handle more complex or practical tasks.<\/p>\n
What the benchmark measures, and what it misses<\/h3>\n
The pelican-on-a-bicycle test primarily measures a model\u2019s ability to generate coherent and visually plausible images based on a specific prompt. It can reveal how well a model understands and combines elements like animals, vehicles, and actions.<\/p>\n
However, the test misses the mark when it comes to evaluating real-world performance. It doesn\u2019t assess how a model handles ambiguity, reasoning, or complex problem-solving. As one experiment found, some models may perform well on this test but struggle in other, more practical scenarios. This highlights a gap between benchmarking and actual AI quality control.<\/p>\n
How AI Labs Performed in the Pelicanmaxxing Test<\/h2>\nKey findings from the 1,008 SVGs generated<\/h3>\n
The results show that the pelican-on-a-bicycle test is not a strong indicator of real-world AI performance. Across the 1,008 SVGs generated, models often produced images that were technically correct but lacked detail or coherence. For example, some models generated a pelican on a bicycle but failed to render the legs or wings accurately. The test is easy to pass, which makes it a poor benchmark for evaluating AI\u2019s ability to handle complex, nuanced tasks.<\/p>\n
Models struggled with prompts that deviated from the original, such as a whale on a plane or an antelope on a skateboard. These prompts required more creative and accurate output, which many models failed to deliver. The test reveals a gap between how AI labs claim progress and how models actually perform under varied, realistic conditions.<\/p>\n
Which models outperformed others in the test<\/h3>\n
Among the seven models tested, GPT-5.6 Terra and Claude Sonnet 5 consistently produced higher-quality SVGs with better coherence and detail. These models scored higher in the judge\u2019s evaluations, particularly in prompts that were more complex or less similar to the original pelican-on-a-bicycle prompt.<\/p>\n
In contrast, models like GLM-5.2 and DeepSeek V4 Pro struggled with consistency and detail, often producing images that were incomplete or lacked the expected elements. The data suggests that while some models are optimized for this specific test, others are not, highlighting the need for more rigorous AI benchmark testing that reflects real-world scenarios.<\/p>\n The pelican-on-a-bicycle test measures a model\u2019s ability to generate a specific image, but it doesn\u2019t reflect how well it performs in real-world scenarios. For example, models may generate a technically correct SVG of a pelican on a bicycle but fail to recognize the same animal or vehicle in a different context. This highlights a critical gap between creative output and practical performance.<\/p>\n When models are tested with variations, like a cat on a skateboard or an antelope on a plane, the results show that they struggle with prompts that are more complex or less familiar. This inconsistency is a red flag for business leaders who need AI systems that perform reliably across a range of tasks.<\/p>\n AI labs are often optimizing for benchmarks that are easy to score and hard to fail. The pelican-on-a-bicycle test is one such benchmark. Labs may be tuning models to perform well on this specific prompt, even if it doesn\u2019t translate to meaningful improvements in real-world applications.<\/p>\n As Simon Willison\u2019s test shows, labs may be \u201cpelicanmaxxing\u201d, focusing on a narrow benchmark, rather than building models that can handle the complexity and variability of actual business use cases. This can lead to inflated perceptions of AI progress that don\u2019t reflect real-world value.<\/p>\n AI quality isn\u2019t measured by how well a model draws a pelican on a bicycle. It\u2019s measured by how it performs in the tasks that matter to your business. For example, if your quality management system relies on AI to detect defects in manufacturing, the real test is whether the model can identify subtle flaws in product images, not whether it can generate whimsical SVGs.<\/p>\n Focus on metrics that align with your operational goals. This includes accuracy in defect detection, consistency in data interpretation, and the ability to handle real-world variability. Tools like GPT-5.6 Luna or Gemini 3.1 Flash-Lite can be used to evaluate these aspects, but only if the prompts and scoring criteria are aligned with your business needs.<\/p>\n Viral benchmarks like the pelican-on-a-bicycle test are easy to pass, but they don\u2019t reflect the complexity of real-world AI applications. A model that excels in these tests may struggle when deployed in a production environment where precision and reliability are non-negotiable.<\/p>\n Business outcomes, such as reduced manual work, faster quality inspections, and higher yield rates, are what matter. These outcomes are what drive ROI. If AI labs are optimizing for viral benchmarks instead of real-world use cases, it\u2019s a red flag. The real test of AI progress isn\u2019t in SVGs, but in the measurable impact it has on your operations.<\/p>\n![\"A](\"https:\/\/falcoxai.com\/main\/wp-content\/uploads\/2026\/07\/are-ai-labs-pelicanmaxxing-th-inline-2.jpg\")
Why This Benchmark Might Be Misleading for Business Leaders<\/h2>\n
The gap between creative output and practical performance<\/h3>\n
What AI labs are really optimizing for<\/h3>\n
What AI Quality Really Means for Your Business<\/h2>\n
Practical AI quality metrics for business use cases<\/h3>\n
Why business outcomes matter more than viral benchmarks<\/h3>\n
![\"A](\"https:\/\/falcoxai.com\/main\/wp-content\/uploads\/2026\/07\/are-ai-labs-pelicanmaxxing-th-inline-3.jpg\")