Language models guess drugs by name suffix, not medical knowledge

People are bringing health and drug questions to language models at increasing rates, often naming a specific medication. New research shows the model can sound knowledgeable while knowing almost nothing about the drug itself. Many drug names encode their pharmacological class through suffixes like -pril, -olol, or -azoline, giving the model a shortcut: infer the answer from the name rather than from learned facts about the drug.

Kaijie Mo of the University of Texas at Austin, with researchers from Northeastern University and MD Anderson Cancer Center, led the study. The team chose Olmo 3 7B Instruct because the analysis required a fully open model — public weights, visible training corpus, documentation from pre-training through post-training, and released intermediate checkpoints. "The openness let us connect behavior, mechanism, and training data," Mo explains. Olmo's corpus includes substantial medical content, making it especially suitable for this test.

The researchers built a diagnostic tool that isolates three possible answer sources: the suffix (the shared ending like -pril), the stem (the unique part of each drug), and genuine knowledge about a specific drug. They swapped the stem, the suffix, or both with meaningless "nonce" strings and measured how answers changed. The method reveals when the model leans on orthographic pattern and when it retrieves an actual fact.

In the relevant evaluation format, roughly 12%-18% of drugs appeared suffix-driven. But the more striking figure: 51%-59% of drugs showed no meaningful sign of specific knowledge — the answer to a real name was nearly identical to the answer to a fabricated one. Searching the open training corpus with infini-gram, a large-scale exact-phrase search engine, showed that the rarer a drug in the training data, the more its answers relied on the suffix and the less on concrete knowledge.

Reliance on suffixes is not a problem in itself; drug suffixes do encode real pharmacological information. The problem arises when useful generalization substitutes for knowledge of the particular drug. For Mo, the value of Olmo 3 and infini-gram lies in the ability to link model outputs to their probable cause. "In high-stakes domains like medicine, auditing outputs alone is often not enough," she says. "Access to the model's internals enables more rigorous diagnosis, validation, and ultimately more trustworthy systems."