Nvidia releases open voice-to-voice model that listens while speaking but breaks after a few turns
Nvidia released NemotronLabs VoiceChat 11B, an open end-to-end real-time speech model that performs continuous streaming of both understanding and generation on a single unified network, without chaining separate ASR, LLM and TTS components. This removes model timing and API hand-offs required by a cascade and shortens end-to-end latency. The model listens while speaking, so a user can interrupt it mid-turn and the agent yields, with a full-cut ratio (TOR 1.00) of 480 ms. It is also the first open full-duplex model that supports tool calling while the conversation continues to flow, using a separate output channel for <TOOLCALL> scripts together with pre-defined “wait” lines that the system inserts while the API runs.
The architecture is a hybrid Mamba/Transformer, built from three existing Nvidia components plus a new output path. The speech encoder is Fast Conformer from Nemotron-Speech-Streaming-En-0.6b, encoding a continuous 16 kHz incoming audio stream. The language-model backbone is Nemotron Nano v2, which consumes audio tokens and predicts text tokens. The TTS decoder predicts audio codes that are rendered as agent speech at 22.05 kHz, and a separate output channel is dedicated to tool-calling scripts. Outputs include the agent’s audio, the agent’s text, and a running transcription of the user. Training used roughly 550 000 hours of audio from real and synthetic corpora, relying on the SALM-Duplex and Audio Flamingo 3 models.
Tool calls are emitted on the side channel as a <TOOLCALL> block, and user code returns results in a <TOOL_RESPONSE> block. The interesting part is the wait message: for each tool, the operator defines a sentence that the agent says the moment the model generates the text that triggers the call, so the conversation does not pause while the API runs. Explicit limits: Nvidia recommends a maximum of five tools per session, the model cannot reliably invoke multiple tools simultaneously, and the user cannot interrupt the agent while a tool is executing. System prompts and tool responses must be ASCII-only and TTS-friendly.
On the Full-Duplex-Bench 1.0 metric the model measured a smooth-turn latency of 448 ms with TOR 0.82, user cut-off at TOR 1.00 in 480 ms, and handling of interruptions with TOR 0.153 (synthetic) and 0.255 (Candor), where lower is better. On the AU Harness BFCL-v3 metric for voice tool calling the model recorded 58.5 % for simple calls, 62.5 % for multiple calls, 42.5 % for parallel calls, 27.5 % for parallel-multiple calls, 89.6 % for irrelevancies and an average of 56.1 %. In Full-Duplex-Bench v3 it achieved 82.5 % tool selection, 44.2 % argument accuracy and 33 % pass@1. Nvidia reports the model ranks second.