PerceptionBench Unveils AI Visual Blind Spots: GPT and Kimi Image Recognition Accuracy Under 60%

When the Strongest AI Still “Misreads” Images: The Visual Reality Shock from PerceptionBench

We often have an illusion that since today’s large language models can even write complex code, understanding an image should be a piece of cake. But the truth is quite the opposite. When you ask top models like GPT or Kimi to perform basic image recognition, they are often just “guessing blindly.” To break this illusion that “AI vision is already flawless,” the Kimi team (Moonshot AI) recently released a visual perception evaluation tool called PerceptionBench. This tool directly exposes the collective dilemma current multimodal models face when understanding the physical world.

Why Didn’t We Discover This Problem Before?

The key lies in how previous visual evaluation (VQA) frameworks bundled “understanding the screen” with “logical reasoning.”

For example, if you show AI a blurry photo of an apple tree and ask where the apples are, even if it cannot see the pixels at all, it can infer that “apples grow on trees” based on language common sense accumulated in its training library. This speculative answering strategy perfectly masked the fact that it actually “couldn’t see clearly.”

This “poor eyesight” defect might seem harmless in a laboratory, but if placed in reality—for instance, in a robot in a warehouse that needs precise object grasping, or an autonomous driving system—a single pixel-level recognition mistake could lead to severe physical collision or efficiency disasters.

PerceptionBench’s approach is to thoroughly strip away reasoning capabilities and focus on testing the purest, most fundamental “atomic perception capabilities.” By designing counter-intuitive scenes, it cuts off the AI’s retreat of relying on language logic to cheat.

Test Results: The 60% Accuracy Ceiling That Can’t Be Crossed

When it is no longer possible to “guess,” even the currently strongest models cannot cross the 60% passing line in pure visual perception tests.

On the PerceptionBench leaderboard, GPT-5.6-Sol scored only 59.7%, followed by Kimi-K3 at 58.5%, and Claude-Fable-5 at 57.2%. With the crutch of reasoning taken away, all top models have an error rate exceeding 40%.

Image source: https://www.kimi.com/blog/perception-bench

This performance manifests as “instability” in actual use. If you repeatedly ask the same model about the same image, its answers are often contradictory—saying there are five people in the picture one second, and changing its mind to six the next. This shows that the AI has not established a solid visual neural foundation; many times, its correctness is just due to luck.

How Was This Test Designed?

After analyzing the failure cases of existing models across 40+ visual tests, the R&D team compiled 3,000 real samples. It covers ten fundamental perception categories:

• Space & Localization: Judging the distance, occlusion, and relative relationships of objects (crucial for robot arm grasping).
• Details & Text Recognition: Fine-grained feature extraction, OCR text recognition, and counting.
• Relationship & Comparison: Visual relationships, attribute comparisons, and context integration.
• Hallucination Test: Testing whether the AI sees objects that don’t exist at all.

In these questions, the AI must answer purely by “looking,” completely unable to rely on external common sense to infer.

Image source: https://www.kimi.com/blog/perception-bench

Why Did AI Become a “Smart Blind Person”?

The problem lies in our over-reliance on “stacking parameters” and “language logic” in the past.

Today’s multimodal large language models (MLLMs) rely heavily on rich linguistic prior knowledge during training. This makes increasing model parameters extremely helpful for improving logical reasoning, writing code, or writing articles; but when facing pure visual tasks requiring fine spatial recognition, counting, or 3D structure, they essentially rely on language logic to “speculate” about the screen, rather than truly “seeing.”

This configuration of “advanced brain, blurry eyes” brings great risks in the real world. If AI Agents possess high permissions in an operating system but lack precise environmental visual perception, even an atomic-level visual deviation, such as misidentifying “2 batteries” as “3 batteries,” could turn an automated task into a catastrophe.

Several Questions Developers Care Most About

• Why does the model give inconsistent answers when I repeat the same question? Because many models are often just “guessing” rather than truly perceiving when answering basic visual questions. Lacking a solid foundation for underlying visual feature extraction, their output results naturally cannot remain consistent across multiple questions.
• What is the biggest difference between PerceptionBench and other tests? Its biggest difference is being “anti-cheat.” Its 10 categories and 3,000 questions are all extracted from real failure cases of existing models in 40+ visual tests. It forces models not to rely on common sense reasoning to cover up visual defects, testing their truest atomic perception strength.
• What impact does this have on future AI development? It forces major laboratories to rethink evaluation and training strategies. PerceptionBench provides a precise diagnostic tool. Future multimodal model development must squarely face and solve these underlying visual weaknesses to build truly faithful and consistent visual AI systems.

Conclusion: From “Guessing” to “Seeing”

This evaluation is not just a leaderboard for ranking; it’s more like a safety boundary.

Only when we stop being superstitious about “larger model parameters mean knowing everything” and divert a portion of R&D resources to strengthening the extraction of underlying visual features will AI have a chance to truly open its eyes. To stand firm in the physical world, having a smart brain is not enough—AI must first learn to “watch the road properly.”

Q&A

Q1: Since the AI’s visual accuracy is less than 60%, why do we still feel they are smart when reading images with ChatGPT or Kimi? A: This is because the images and questions we usually input often allow AI to compensate for visual deficiencies through “language logic” or “common sense.” For example, when AI dimly sees a kitchen background and a figure, even if it can’t clearly see the specific tools held in hand, it can “speculate” that the person is cooking. The severity of PerceptionBench lies in that it deliberately cuts off all retreats that could rely on common sense, forcing the model to give answers purely by “looking (atomic perception),” which is what exposes the underlying “vision defects.”

Q2: What specific “atomic perception” capabilities does PerceptionBench test? A: The evaluation’s 3,000 questions were not fabricated out of thin air, but extracted from real failure cases of existing models across 40+ visual tests. It covers 10 of the most overlooked fundamental visual foundations, including: Visual Relation, Counting, Attribute, Depth & 3D Perception, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR Text Recognition, and Hallucination tests.

Q3: Which models are currently performing best on the leaderboard? Is there a large gap in scores? A: According to the official latest leaderboard, the current top three are GPT-5.6-Sol (59.7%), Kimi-K3 (58.5%), and Claude-Fable-5 (57.2%). Although they are recognized as the strongest multimodal models in the industry, the scores are not only very tight, but they all fall below the passing line (60%). This shows that “poor visual perception capability” is not a shortcoming of a single vendor, but a collective underlying architectural bottleneck currently faced by the entire AI industry.

Q4: If my enterprise project needs to introduce multimodal AI to process images (such as automated reporting, visual inspection), what practical advice do you have? A: According to the phenomenon revealed by PerceptionBench: today’s models are often “blindly guessing based on luck” on many issues, resulting in inconsistent answers when the same image is repeatedly questioned. Therefore, when applied to critical tasks (such as medical imaging assistance, industrial defect detection, or fine-grained physical robot operations), multimodal large models must absolutely not be used as a single decision node. It is recommended to use double verification with traditional computer vision algorithms (such as specialized object detection models), or maintain a “Human-in-the-loop” review mechanism until this 60% visual ceiling is thoroughly broken.