Nvidia launches Switchyard: Rust proxy for routing LLM traffic between OpenAI and Anthropic

The problem is familiar to any team that runs code agents: Claude Code talks to the Anthropic Messages API, Codex CLI talks to OpenAI, and the model the team wants to run sits behind vLLM, Nvidia NIM or Ollama. Re-writing the agent is not an option, so the translation layer must sit elsewhere. Switchyard is Nvidia’s answer – a proxy and Rust library that routes requests between providers, translates between OpenAI and Anthropic formats, records operational metrics and exposes modular routing algorithms. The code is released under the Apache 2.0 license with documentation at docs.nvidia.com/nemo/switchyard.

What Switchyard actually does

Clients keep their original API. Switchyard parses an incoming request into neutral Rust types for the provider, runs a routing algorithm that selects a backend, re-encodes the request in the wire format of that backend, calls it, and translates the response – including streaming events – back into the shape the client expects. The server accepts three incoming formats: OpenAI Chat Completions, OpenAI Responses and Anthropic Messages. Each of the three can target any backend, and each LLM client is configured with its own preferred top-level format. The decoupling is the key point: the agent API and the backend API no longer have to match.

Three ways to run the tool

*Launcher path* targets code agents. Install the popular tool with

```

uv tool install --python 3.12 "nemo-switchyard[cli]"

```

then run `switchyard launch claude`, `switchyard launch codex` or `switchyard launch openclaw` against a packaged deployment or your own TOML file.

*Server path* installs the standalone proxy with

```

cargo install --locked switchyard-server

```

validates configuration with `--dry-run`, and listens on a host and port of your choosing.

*Library path* uses `switchyard-libsy`, which embeds the routing algorithms in a Rust application without maintaining its own HTTP stack; the algorithm decides where to route and returns each model call to the caller.

Routing algorithms: from passthrough to stage_router

A route is a client-visible model identifier plus the algorithm behind it. The server supports four algorithms:

* `passthrough` forwards every request to a single destination;

* `random` distributes traffic among destinations with optional relative weights and an optional seed for reproducible selection – the path for A/B experiments and cost testing;

* `llm_classifier` queries a classified target for a capability judgment, then routes to a weak or strong destination based on `base_threshold` and parameters such as `min_confidence`, `capability_elevated_floor` and `session_affinity`; if the classifier cannot decide, the request falls back to the strong destination;

* `escalation` runs each queue on the weak tier first and lets the classifier decide whether to re-run on the strong tier;

* `stage_router` scores tool-output signals and agent-progress signals from later queues to pick a capable or efficient destination, avoiding a classifier call in most queues. Roles such as strong, weak, capable and efficient are defined within a route, not as fixed model attributes, so the same top-level model can assume different roles in different routes.

Observability and metrics

`GET /metrics` returns Prometheus-format text supplied by OpenTelemetry at the process level. Metric families cover request counts, errors, model-call latency, full-queue latency, prompt tokens, completions, cache hits, cache creations, inference, and top-level HTTP attempts broken down by result and code. The label `tier` carries `strong` or `weak` for classifier-derived decisions, and classifier calls are excluded from those families. The most interesting metric is `switchyard_routing_overhead_ms`, which reports the runtime of the routing algorithm itself, giving teams a view of how much routing costs before any model is invoked.

Pre-alpha warning: not for production

The tool can be deployed today – the binary is available from crates.io, the launcher from PyPI, and it self-hosts anywhere – but Nvidia marks Switchyard as pre-alpha and experimental, warns that it is not for production use, and states that the API and algorithms are expected to change significantly before version 1.0. Teams building agent infrastructure and wanting to experiment with cross-provider routing without writing translators themselves can start playing now; teams that need contractual stability will have to wait.