Andrew Feldman says Cerebras wafer-scale architecture runs inference 2,500 times faster than GPU

Andrew Feldman, co-founder and CEO of Cerebras, attributes the performance gap to the way large language models execute inference. Inference consists of two distinct phases: pre-fill, in which the model processes the user's prompt, and decode, in which it generates the response token by token. The decode phase creates the significant bottleneck.

During each decode step, before computing the next token, the model’s weights must be moved from memory to the compute units. This transfer repeats for every token, so the speed of that movement determines the overall output rate. In conventional graphics hardware, weights reside in high-bandwidth memory (HBM) that is physically close to the GPU, yet accessing it still requires data to travel over an external interface.

Cerebras adopts a different approach: its wafer-scale processor spreads across an entire silicon wafer, and the weights are stored in SRAM that is distributed directly alongside the compute units. SRAM operates at speeds orders of magnitude higher than HBM, and the physical proximity eliminates the need to shuttle data over long bus distances. Feldman states that this memory-to-compute movement is 2,500 times faster than on a GPU.

The 2,500× figure refers specifically to the weight-transfer rate per token in the decode stage, not to overall model performance, not to the pre-fill stage, and not to training. The acceleration matters when a model is too large to fit in a GPU’s cache and inference is memory-bound; in other scenarios the gap narrows significantly.

Feldman made these remarks in an episode of The MAD Podcast with Matt Turck and on Cerebras’ YouTube channel. As is typical in hardware vendor statements, the numbers describe optimal conditions and a theoretical architecture; production performance depends on model size, context length, the software stack, and batch size. Independent measurements will be needed before drawing broad conclusions.