Large language models ace benchmarks but fail at structuring knowledge, study finds

A new preprint on arXiv shows that large language models — open and closed alike — do not organize mathematical knowledge in a way that resembles human learners. The researchers used Knowledge Space Theory (KST) as a normative framework: under the theory, mastering a concept requires prior mastery of its prerequisites, and any deviation from that order signals an incoherent knowledge structure. The result: models violate knowledge dependencies at high rates and fail to exploit related knowledge supplied in context to improve performance on dependent questions.

The framework: Knowledge Space Theory as a lens

KST was originally formulated to map human knowledge in domains such as mathematics. It defines a "knowledge space" as a collection of possible knowledge states, each closed under prerequisite relations. The researchers built an evaluation protocol on this foundation that checks whether a model's answers are consistent with that structure — not just whether the answer is correct, but whether the pattern of correctness and failure across a sequence of questions matches the dependency hierarchy. This is a fundamental departure from standard benchmarks that count only final accuracy.

The experiment: eight models versus real learners

The team ran the protocol on eight models, some open-source and some proprietary, and compared the resulting distributions to those of human learners on the same tasks. The most striking first finding: there is little overlap among the distributions of the different models. Each model "invents" its own internal knowledge structure, and there is no consensus among them on what depends on what. In other words, they not only differ from humans; they differ from each other.

Dependency violations and failure to leverage context

When the researchers supplied models with relevant knowledge in context — for example, a reminder of a definition or a prerequisite theorem — the models did not translate that into consistent improvement on questions that depended on that knowledge. Humans, by contrast, exploit such reminders systematically. The violation of knowledge dependencies appears even when the model knows the answer to the basic question but fails on its derivative, a phenomenon KST defines as impossible in a coherent knowledge space.

The blind spot of accuracy metrics and automated judging

More troubling: both accuracy-based evaluation and LLM-as-judge evaluation fail to detect the structural failure. A model can score highly on a standard benchmark while holding a completely broken knowledge structure. Automated judges, trained on human preferences, tend to reward answers that sound plausible without checking structural consistency. This gap explains why the phenomenon has remained invisible until now.

What this means going forward

The practical takeaway: high performance on reasoning tasks does not indicate structural understanding. If we want models that behave like experts rather than merely answering like experts, we need metrics that test knowledge coherence, not just pointwise correctness. The study is still a preprint and has not undergone peer review; the sample is limited to eight models and a single mathematical domain. But the proposed framework is extensible and gives a clear direction for where to look in future evaluations.