Transformer models face soaring costs, prompting search for alternatives

The transformer, now nine years old, is beginning to show its age. Since researchers at Google published the 2017 paper “Attention Is All You Need,” which introduced a new neural-network architecture for text processing, the transformer has become the engine hidden beneath every major large language model (LLM) on the market. Justin Dangel, chief executive officer and co-founder of the startup Subquadratic, says the entire industry is built on this architecture, calling it one of the most important inventions in the history of computer science. Yet the structural weaknesses of these models can no longer be ignored.

The core issue lies in the mechanism that gives the transformer its advantage: dense attention. Dense attention encodes the meaning of text by comparing each word—or token—to every other token in the same text through multiplication. Accuracy is high, but the computational cost escalates rapidly. A document of 10,000 words forces the model to perform roughly 50 million multiplication operations, which is the primary driver of the enormous electricity consumption of LLMs.

The economic price of this limitation is staggering. Greg Brockman, president of OpenAI, says the company expects to spend 50 billion dollars on compute resources this year. The International Energy Agency (IEA) also projects that data-center electricity use will double by 2030. Companies attempting to run models on massive data sets encounter difficulty maintaining a context window wide enough, because the word-by-word processing inherent to the transformer limits how much context can be retained.

Recent advances such as reasoning models and the ability to process large inputs in parallel are not natural extensions of the core technology. They function as patches designed to work around structural flaws. Reasoning models operate by writing “chain of thought” notes to a draft notebook and then rereading them, further burdening data management. As models grow and become more sophisticated, the transformer becomes a bottleneck, and the technology’s principal strength becomes a genuine limitation.

A wave of startups is now pushing the boundaries and proposing alternatives. While many will inevitably fail, they have far less at stake than the incumbents that dominate the market. One clear strategy for reducing cost and speeding up models is to attack the problem directly by altering the attention mechanism. Replacing dense attention with sparse attention—computing only a subset of token-pair interactions—can dramatically cut the number of required calculations. Researchers have produced a variety of sparse-attention mechanisms over the years, and whether they will ultimately supplant classic transformers remains an open question.