Andrej Karpathy proposes replacing SVG tests with full code-generated worlds

Karpathy posted a new benchmark for AI models: instead of asking a model to “draw a pelican riding a bike in SVG”, he asked Opus 5 to turn a paragraph from *The Lord of the Rings* into a complete interactive scene built with Three.js. The result was 5,500 lines of code that run a graphical world in a browser, including lighting, textures and animation. The generation took about two hours, consumed roughly one million tokens and cost around $10. The accompanying audio was supplied by an ElevenLabs model.

Karpathy argues that traditional spatial-reasoning tests, such as the SVG pelican task, have become trivial for modern multimodal models. The limitation, he says, is not the models’ drawing ability but what the tasks measure: they evaluate a static, one-off output rather than the capacity to construct a living system that responds to input, preserves state and self-corrects over time.

His alternative benchmark is simple: assess a model by its ability to generate code that runs an entire world—physics, rendering, game logic, user interface—from a brief natural-language description. This test blends language comprehension, architectural planning, clean coding within a real ecosystem (Three.js, WebGL) and iterative bug fixing while the program runs. In other words, it demands full-stack software engineering, not just localized code completion.

The main bottleneck today is output quality, because the model cannot analyze the video it produces. It must rely on static screenshots to spot rendering errors, inverted textures, misplaced objects, depth issues and then correct them in subsequent iterations. Without a continuous visual feedback loop, fixes are slow, partial and vulnerable to error accumulation. When models gain direct access to their own frame streams, a sharp improvement in world quality is expected.

If this benchmark is adopted, it could push the industry toward models that understand not only “what an object looks like” but also “how a system behaves”. That would raise the bar from one-shot generative ability to systemic engineering capability—the kind required of agents that must write, execute and maintain code in real production environments. Opus 5 demonstrated that the approach is feasible; the open question is how quickly video self-analysis will arrive to make it reliable.