LLM Agent Midterm Exam: VitaBench Reveals Harsh Truth, Top Models Only 30% Success Rate?

Just when we thought AI agents driven by Large Language Models (LLMs) were omnipotent, the latest benchmark VitaBench, released by Meituan’s LongCat team, serves as a reality check for the entire industry. This “hardest mock exam” shows that even top AI models have a surprisingly low success rate when dealing with complex real-world tasks. What is going on?

When AI Agents Step Out of the Lab, Reality Hits Hard

In recent years, AI agents driven by Large Language Models (LLMs) have undoubtedly been the hottest topic in the tech world. We imagine a future where, with just a few words, AI assistants can handle everything from booking restaurants and planning trips to arranging deliveries. Sounds great, right?

But reality is often a bit harsh. Current AI agents may perform well in simple, closed environments, like driving in a training course—everything goes smoothly. However, can they still cope when placed at the crossroads of the real world—a complex environment full of unexpected situations, vague instructions, and multiple tasks?

The answer might be a little disappointing. Many past evaluation benchmarks have oversimplified problems and failed to truly reflect the complexity of real life. It’s like using a linear equation to assess a mathematician’s ability—it doesn’t measure their true skills at all.

VitaBench: The “Ultimate Proving Ground” for AI Agents

To solve this problem, Meituan’s LongCat team launched VitaBench—a new, high-difficulty benchmark designed specifically to evaluate the performance of LLM agents in real-world applications.

You can think of VitaBench as an extremely realistic “life simulator.” It’s no longer about theory; it throws AI directly into the three major life scenarios we are most familiar with:

• Food Delivery
• In-store Consumption
• Online Travel Services

How complex is this simulated environment? It integrates up to 66 different tools, covering almost all possible operations from querying store information, making reservations, and placing orders to payment.

Not Just Single Tasks, but a Continuous “Cross-Scenario” Challenge

The core challenge of VitaBench lies in its task design. It not only has 300 single-scenario tasks but also 100 extremely challenging “cross-scenario tasks.”

What does this mean? For example, a real user request might be: “Help me book a hotel with a river view, and for the night of check-in, find a well-rated, non-spicy restaurant near the hotel with a budget of $200.”

This task requires the AI agent to:

1. Understand Complex Intent: Not just booking a hotel, but also a restaurant, and the two are related.
2. Cross-Temporal and Spatial Reasoning: Needs to handle check-in dates, dinner times, and the geographical relationship between the hotel and the restaurant.
3. Flexible Use of Tools: Must first use the “hotel booking tool” and then use the “restaurant search tool” based on the results.
4. Proactive Clarification: If the user’s instructions are vague, the AI needs to ask follow-up questions, such as “What type of cuisine would you prefer for the restaurant?”
5. Track Dynamic Intent: In a multi-turn conversation, the user might change their mind, and the AI needs to keep up.

Honestly, this is a bit complicated even for humans, let alone AI.

The Brutal Report Card: Top AIs “Fall” One After Another

So, how did the most powerful AI models of today perform in this ultimate test?

The results are quite shocking.

Thinking Models

Non-thinking Mode

According to the Leaderboard published by VitaBench, the data reveals a huge performance gap:

• In the relatively simple 300 single-scenario tasks, even the best-performing models have a success rate of less than 50%.
• In the 100 complex cross-scenario tasks, the success rate of the strongest models plummets to just 30%!

This report card clearly tells us that current LLM agents have significant shortcomings in the following areas:

• Difficulty in Domain Switching: An AI that is good at handling travel bookings can easily “crash” when asked to handle dining issues at the same time.
• Tool Selection Obstacles: Faced with 66 tools, AI often doesn’t know when and which one is the most appropriate to use.
• Lack of Long-term Coordination: Handling long-term tasks that require multiple steps and span several rounds of conversation remains a huge challenge for AI.

What Does This Mean for Our Future?

The emergence of VitaBench is not meant to undermine our confidence in AI. On the contrary, it acts like a mirror, truthfully reflecting the current technological deficiencies and pointing the way forward for the entire industry.

This research tells us that to make AI agents truly reliable assistants in our lives, we must not only focus on improving the language capabilities of the models but also train their ability to reason, plan, and execute tasks in complex, dynamic environments.

VitaBench provides a valuable resource for developers to test and improve their AI agents in an environment closer to reality. Although the current 30% may seem low, this is precisely the stage of accumulating strength before a technological take-off.

Frequently Asked Questions about VitaBench

Q1: What exactly is VitaBench? A: VitaBench is a high-difficulty evaluation benchmark developed by the Meituan LongCat team, specifically designed to assess the ability of Large Language Model (LLM) agents to perform complex interactive tasks in simulated real-world scenarios (such as delivery, travel).

Q2: Why do we need evaluation tools like VitaBench? A: Because existing evaluation tools are mostly oversimplified and cannot reflect the complexity of real-world tasks. VitaBench provides a “testing ground” that is closer to reality, which can effectively test the true capabilities of AI agents in handling multiple goals, dynamic information, and complex toolsets, thereby promoting the practical application and development of the technology.

Q3: Which AI models are currently performing best on VitaBench? A: According to the published leaderboard, in the most challenging cross-scenario tasks, models such as o3 (high), Claude-4.1-Opus (w/ thinking), and LongCat-Flash-Thinking are in the lead, but even so, their highest average success rate is only around 30%.

Q4: How can I learn more about or use VitaBench? A: The VitaBench project is open source. You can visit its official website to view the detailed research paper, dataset, and leaderboard. Developers can also find the relevant code and resources on its GitHub page.