First-of-its-kind study finds simple agents succeed in production

Researchers presented at ICML 2026 the first systematic study of Large Language Model (LLM) agents in real-world production settings. The study, named MAP (Measuring Agents in Production), is based on 20 test cases collected through in-depth interviews with developers and on a survey of 86 team members who operate such systems across 26 different domains. Its aim was to understand why organizations build agents, how they construct them, how performance is evaluated, and where the main development challenges lie.

The data paint a picture that differs from common laboratory assumptions. 68 percent of production agents perform no more than 10 steps before a human intervenes. 70 percent rely on prompting off-the-shelf models without any weight fine-tuning. 74 percent primarily use human evaluation as the main performance metric. In other words, the systems that work in practice are those that deliberately limit autonomy rather than those that try to be “smart”.

Developers report reliability as the top challenge – the ability of an agent to behave correctly consistently over time, not just in a single example. The industry-standard solution is not a larger model or a more sophisticated architecture, but system-level design: hard limits on step count, built-in control mechanisms, and predefined human-intervention points. This is intentional restrictive engineering rather than an algorithmic breakthrough.

MAP documents the current state and highlights gaps that academia has largely ignored: trustworthy automatic evaluation that could replace manual checks, methods for improving consistency without sacrificing flexibility, and a clearer understanding of when weight fine-tuning is worth the effort versus simple prompting. The researchers hope that this visibility will steer the scientific community toward problems that truly trouble production-run agents, instead of chasing benchmarks detached from reality