VLLM offloads video decoding to NVDEC, doubles throughput on 8×H100

The vLLM inference library has integrated PyNvVideoCodec and moved video decoding from the CPU to NVDEC, Nvidia's hardware decoder, removing a bottleneck that limited scaling of video-captioning workloads on multi-GPU nodes. The change delivers more than 2× throughput on an 8×H100 configuration and is already included in the standard CUDA builds of vLLM.

The problem: CPU choked before the model even started

Until now vLLM relied on an OpenCV- and FFMPEG-based CPU backend. In vision-language models that produce relatively short outputs — 100 to 200 tokens per caption — decoding accounts for an outsized share of total latency. On nodes with two to four GPUs the CPU was already fully saturated, and every additional GPU only made the contention worse. For organizations processing hundreds of thousands of video hours to train autonomous vehicles, it was a hard ceiling.

The fix: PyNvVideoCodec wraps NVDEC directly

PyNvVideoCodec is a Python wrapper around Nvidia's hardware decoders. The integration lets vLLM shift the decode workload onto NVDEC, freeing CPU cores for inference itself. In the standard CUDA releases of vLLM the library is already bundled (PyNvVideoCodec==2.0.4); custom installations need to add it manually as a dependency.

Real scaling: from 2× to 8×H100 without breaking a sweat

In Nvidia's tests on autonomous-vehicle captioning tasks, the move to GPU decoding yielded more than 2× throughput on 8×H100 versus the old backend. The scaling curve stays nearly linear out to eight replicas — one per GPU — while the CPU-based version flattens between two and four cards. For AV teams chewing through hundreds of millions of clips, that translates to entire processing days saved.

In production: MPS, VRAM reservation, and single-GPU containers

To realize the gains you must start the CUDA MPS Daemon before `vllm serve`; it's mandatory for multi-process inference workloads. The `--mm-ipc-gpu-memory-gb` flag reserves VRAM for decoding — test different values and keep the minimum that doesn't hurt throughput. The recommended architecture: a separate container for each vLLM replica pinned to a single GPU (via `CUDA_VISIBLE_DEVICES`), with a reverse proxy distributing requests.