Nvidia launches Cosmos-H-Dreams, a real-time generative simulator for surgical robotics

Nvidia announced Cosmos-H-Dreams, a conditional-action generative simulator that runs in real time on a single RTX PRO 6000 GPU. The system distills the capabilities of Cosmos-H-Surgical-Simulator into a causal student model with a few steps and delivers it through FlashDreams, Nvidia’s accelerated inference library for streaming. The result is an interactive environment that a human or a learned policy can control in a closed loop.

Cosmos-H-Surgical-Simulator, the foundation of the new development, is a world foundation model built on Cosmos-Predict2.5-2B and post-trained on the Open-H-Embodiment dataset. Given a surgical context frame and a future robot trajectory, it generates video of the expected visual consequences of those actions, useful for offline policy evaluation and synthetic data creation. Cosmos-H-Dreams extends this capability to real-time operation: starting from the multi-body surgical priors learned by the original model, it specializes in tabletop suturing with the da Vinci Research Kit (dVRK) and refines the behavior into a causal student model that produces the scene autoregressively. The released model receives an initial RGB frame and a live stream of robot kinematics, then generates the next frame segment before proceeding to the next action block.

The flexibility of Cosmos-H-Dreams was demonstrated in collaboration with CMR Surgical and Cambridge Consultants, who integrated it with the Versius surgical controller, enabling real-time execution on the Versius platform. This marks a step beyond the academic dVRK system toward an active commercial platform.

A central challenge is preserving useful surgical dynamics while reducing generation cost. Cosmos-H-Dreams employs a teacher-student pipeline designed for long autoregressive rollouts. The bidirectional teacher starts from the open checkpoint of Cosmos-H-Surgical-Simulator on Open-H, which uses a unified 44-dimensional action representation. For the released tabletop dVRK model, the dual-arm action content, a relative translation of the end-effector, rotation, and gripper state are mapped to this shared representation.

The teacher undergoes fine-tuning on a mixed dVRK tabletop dataset from Johns Hopkins University (JHU) that includes successful demonstrations alongside failure segments and out-of-distribution cases such as needle drop, missed injections, and unsuccessful knot tying. These failures are essential: a simulator intended for policy evaluation must reproduce the consequences of poor actions, not only ideal demonstrations. To improve stability in long rollouts, the training gradually expands the teacher’s horizon, starting at 12 frames and increasing to 72 frames, with each expansion re-initializing the warmed-up model using pre-trained weights.

Denoising trajectories computed by the teacher are pre-computed and cached. The causal student is initialized from the teacher and trained to imitate the stored trajectories. This warm-up stage teaches the student to operate with causal attention and a streaming key/value cache before it begins learning from the history it generates itself, addressing the known issue of autoregressive models that see clean ground-truth context during training but must cope with accumulating errors at inference time.