Prime Intellect launches Prime Inference: serving frontier open models on Blackwell

Prime Intellect is releasing Prime Inference, a serving layer for frontier open models that runs on its own GPU fleet across multiple data centers. Before the public launch the system was already processing nearly 1 trillion tokens a day for internal workloads — RL rollouts, synthetic data generation, evaluations, and long-horizon coding agents. The layer closes the loop on an open training stack that already includes prime-rl, verifiers, and sandboxes: deployed models produce production traces that feed back into training. The company says its GLM-5.3 endpoint ranks among the fastest on OpenRouter, with near-zero tool-call error rates and 100% uptime since going live.

Architecture: two modes, open compatibility

The platform offers two operating modes: serverless endpoints for variable demand and reserved capacity for sustained workloads. The interface is OpenAI SDK compatible, so any code that already talks to OpenAI can be pointed at Prime's API URL without rewriting the client. Availability comes from automatic failover across data centers when a given installation goes down. Current hardware is NVIDIA Blackwell, with Vera Rubin listed as "coming soon." Billing is unified with team-level usage tracking; per-model pricing has not yet been fully published in the docs.

Under the hood: Dynamo, vLLM, Mooncake

The implementation combines NVIDIA Dynamo, vLLM, Mooncake, and FlashInfer, built in collaboration with Inferact and NVIDIA with fixes pushed upstream. The target workload is agentic: a typical agent turn adds roughly 6,000 tokens to a 140,000-token prompt, and benchmarking uses SemiAnalysis AgentX plus synthetic cold starts. Prefill-decode disaggregation runs on separate GPU clusters; Dynamo routes, vLLM runs the model on each cluster, and decoders pull KV cache over NIXL. Prime says this yielded roughly 40% lower p90 inter-token latency in its tests. Cache-aware routing weighs prefix cache overlap against queue depth and keeps sessions on the same decoder across turns; Mooncake adds a second KV tier in host DRAM.

GLM-5.3 numbers on GB200

The interactivity target is 100 end-to-end tokens per second per user. At a 1:4 prefill-to-decode ratio the highest throughput was recorded: 66 sessions per prefill cluster at 101 tok/s/user and 100 tok/s/GPU output. DEP8 prefill topology delivered 5x more usable prefix-cache capacity than TEP8 on the same iron. Cutting the per-step token budget from 8K to 4K per GPU halved median queue wait from 550 ms to 110 ms, and median time-to-first-token fell by roughly 20%. KV compression with NVFP4 shrank an MLA cache line from 576 to 352 bytes, raising tokens in cache per decoder from 1.09 million to 1.63 million. A native-sparse MLA kernel clocked roughly 12.0 microseconds at 15 query tokens, versus 17.7 microseconds for a staged approach and 13.7 microseconds for FP8; the company notes this is workload-dependent. BLHNC KV layout cut transfer descriptors from 19,559 to roughly 1,940 and average transfer time from 146 ms to 78 ms.

Flawless tool calls

Agents fail when tool calls arrive with wrong names or broken arguments. Prime's team contributed a structural-tag builder to Dynamo for GLM's tool format; vLLM uses xgrammar to mask tokens that violate the tool schema. Parsing bugs were also fixed, including a case where `<` was decoded as `<` inside code.

Pricing and roadmap

Competitor prices were verified on October 2, 2026 against ComputePrices, a third-party price tracker. On the map: batch inference and one-click dedicated deploys.