PerceptionBench Exposes AI Visual Blind Spots: Image Recognition Accuracy of GPT and Kimi is Below 60%

When the strongest AI still “sees” images wrong: The visual reality shock brought by PerceptionBench

We often have an illusion that since today’s Large Language Models can even write complex code, understanding an image should be a breeze. But the truth is the opposite. When you ask top-tier models like GPT or Kimi to perform the most basic image recognition, they are often just “guessing.” To break this illusion that “AI vision is already flawless,” the Kimi team (Moonshot AI) recently released the visual perception evaluation tool PerceptionBench. This tool directly exposes the collective dilemma of current multimodal models in understanding the physical world.

Why didn’t we notice this problem before?

The key lies in the fact that past visual evaluations (VQA) bound “understanding the screen” and “logical reasoning” together.

For example, if you show an AI a blurry photo of an apple tree and ask where the apple is, even if it cannot see the pixels at all, it can infer that “apples grow on trees” based on the linguistic common sense accumulated in its training library. This opportunistic answering strategy perfectly masked the fact that it actually “could not see clearly.”

This “poor eyesight” defect might seem trivial in a laboratory, but if applied in reality, such as in logistics warehouses where robots need to accurately grasp objects, or in autonomous driving systems, a single pixel-level recognition error could lead to serious physical collisions or efficiency disasters.

The approach of PerceptionBench is to completely strip away reasoning capabilities and focus on testing the simplest and most fundamental “atomic perception ability.” By designing counter-common-sense scenes, it cuts off the AI’s path to cheat by relying on linguistic logic.

Test results: The 60% accuracy ceiling that cannot be crossed

When it is no longer possible to “guess,” even the most advanced current models cannot cross the 60% passing line in pure visual perception tests.

On the PerceptionBench leaderboard, GPT-5.6-Sol only achieved a score of 59.7%, followed closely by Kimi-K3 at 58.5%, and Claude-Fable-5 at 57.2%. When the crutch of reasoning is taken away, the error rate of top-tier models all exceeds 40%.

This performance is reflected in practical use as “instability.” If you ask the same model about the same image repeatedly, its answers are often contradictory—one second it says there are five people in the picture, and the next it says there are six. This indicates that the AI has not established a solid visual nervous system; often, getting it right is just good luck.

How is this test designed?

After analyzing the failure cases of existing models in more than 40 visual tests, the R&D team organized 3,000 real samples. It covers ten basic perception categories:

• Space and positioning: Judging the distance, occlusion, and front-back-left-right relationships of objects (this is critical for robotic arms grasping objects).
• Detail and text recognition: Fine-grained feature extraction, OCR text recognition, and counting.
• Relationships and comparison: Visual relationships, attribute comparisons, and context integration.
• Hallucination test: Testing whether the AI will see objects that do not exist at all.

In these questions, the AI must answer purely by “seeing” and cannot rely on external common sense to deduce at all.

Why has AI become a “smart blind person”?

The problem lies in our over-reliance on “stacking parameters” and “linguistic logic” in the past.

Today’s multimodal Large Language Models (MLLMs) are highly dependent on rich linguistic prior knowledge during training. This makes increasing model parameters extremely helpful for improving logical reasoning, writing code, or writing articles; but when faced with pure visual tasks that require fine spatial recognition, counting, or 3D structures, they are essentially still “speculating” on the screen based on linguistic logic, rather than truly “seeing.”

This configuration of a “developed brain but blurred eyes” will bring great risks in the real world. If AI Agents have high privileges in operating systems but lack precise environmental visual perception, even an atomic-level visual deviation, such as seeing “2 batteries” as “3 batteries,” could turn an automated task into a disaster.

Several questions that developers care about most

• Why does the model give inconsistent answers when asked the same question repeatedly? Because many models are often “guessing” rather than truly perceiving when answering basic visual questions. When lacking solid fundamental visual feature extraction capabilities, their output results naturally cannot remain consistent across multiple inquiries.
• What is the biggest difference between PerceptionBench and other tests? The biggest difference is “anti-cheating.” Its 10 categories and 3,000 questions are all extracted from the actual failure cases of existing models in more than 40 visual tests. It forces the model not to be able to rely on common sense reasoning to compensate for visual defects, measuring the truest atomic perception strength.
• What impact does this have on the future development of AI? This forces major laboratories to rethink evaluation and training strategies. PerceptionBench provides a precise diagnostic tool. The development of future multimodal models must face and solve these fundamental visual weaknesses in order to build truly faithful and consistent visual AI systems.

Conclusion: From “guessing” to “seeing”

This evaluation is not just a leaderboard to compare who is better; it is more like a safety boundary.

Only when we no longer blindly believe that “if the model parameters are large, it understands everything,” and distribute a portion of R&D resources to strengthening the extraction of fundamental visual features, will AI have a chance to truly open its eyes. To stand firm in the physical world, having a smart head is not enough; AI must first learn to “look at the road properly.”