Decision 3.0 lands on Core ML: semantic routing on Apple silicon

The vLLM Semantic Router team has shipped Decision 3.0 as a Core ML package, and the on-device numbers on an M5 Pro are striking: 0.12 seconds per PR, 8.3 requests per second, all local with zero bytes leaving the machine. The model chews through a thousand PRs and sorts them into vllm-sr work groups — a job that normally demands a cloud round-trip — in a single forward pass on the prompt's last token.

Under the hood

Unlike a regular LLM that generates text, Decision performs 255-way classification in one call. The forward pass returns an FP32 probability vector for every option — no decoding, no sampling, no generation latency. The contract matches upstream exactly: same function signature, same behavior, only the graph is now implemented in Core ML and executes on Apple's Neural Engine. Three sizes — lite, nano, mini — cover text, image, and video in the same repo, each with its own graphs, vision tower, host tables, host Python code, and a README that lists files, a parity check against the original, and a speed benchmark.

Swift integration and licensing

On the client side, D3VisionManager inside FluidUse loads any of the models directly — no extra conversion, no awkward wrappers. The license stays Apache-2.0, so commercial embedding is friction-free. The published figures come from a live run on M5 Pro; no benchmarks exist for earlier Apple silicon generations, so performance on M3 or M4 is unknown without testing locally.

Why it matters for developers

The practical upshot: semantic request routing, sending a question to the right model, intent classification, multimodal segmentation — all can now happen on-device, with no network dependency, no cloud cost, and no data leakage. For apps that need privacy, low latency, or offline operation, this opens a door that was previously shut. Downloads over the past month signal interest, but without official numbers real adoption is hard to gauge.