VLLM adds a specialized decoding engine without breaking existing deployments

The vLLM project has launched an official integration with TileRT, an inference runtime built for per-user decoding speed, through vLLM's open V1 connector interface. The package ships with TileRT 0.1.5 and lets users run prefill in standard vLLM and hand off the decoding phase to the dedicated engine — no core-code changes, no fork, no internal patches. The idea is straightforward: when prefill and decoding are architecturally separated, the decoding side becomes pluggable, and each workload can pick the engine that fits.

Why a second decoding engine

vLLM's original decoder remains the right default: it is built for high aggregate throughput across a wide range of models and hardware. But a growing class of workloads — agentic loops, interactive coding assistants, real-time voice — is latency-bound. The metric that matters there is not how many tokens the system emits in total, but how fast each user gets their next token. TileRT was designed from the ground up for exactly that regime. The two engines sit at different points on the same throughput-latency frontier, so they complement rather than compete.

Architecture: planned coexistence

The guiding principle is zero changes to vLLM. The integration lives entirely behind the public V1 extension surface: an implementation of `KVConnectorBase_V1` composed under `MultiConnector` and loaded through the standard `kv_connector_module_path` mechanism. Beyond engineering aesthetics, this means adding a TileRT decoding pool cannot destabilize an existing vLLM deployment, and a version upgrade does not require re-porting a fork. Lightweight routing sits in front of the TileRT pool: for each request it sets `max_tokens=1` so vLLM performs prefill and emits the first token, then attaches the target node in the standard `kv_transfer_params` pass-through field. Traffic to the original pool flows through the regular disaggregation proxy, unchanged.

Claim filtering and pure producer

The TileRT connector "claims" only requests that carry the dedicated marker and behaves as a complete no-op for everything else, so the two decoding pools can share a single prefill instance — even a single front-end batch — without TileRT adoption for part of the traffic affecting the rest. The connector acts solely as a `kv_producer`: it extracts and ships state after prefill and does not touch scheduling or sampling. In other words, the OpenAI-compatible API, scheduling, prefix caching, tool calling, and vLLM's operational maturity all stay exactly where they are; only the decoding pipe is swapped, and only for callers that ask for it.