Research creation · How actions become video
Move a controller, generate a surgical scene: what does Cosmos3-H-Dreams simulate?
One starting image and different tool actions produce different continuations. NVIDIA turns generated video into an interactive research environment; visual realism, action response and validated tissue mechanics need separate evidence.
This interactive simulator is intended for robotics research and requires Linux with an NVIDIA GPU, with over 32GB VRAM in its default configuration. NVIDIA published the work on October 20. Generated scenes and action responses are worth inspecting; patient use and tissue mechanics require separate validation.
Start with one scene and inspect two continuations
In the public demo, the operator moves controllers while instruments and tissue change on the display. Cosmos3-H-Dreams takes an endoscopic image, a short action history and future tool commands, then generates subsequent video. It explores a move from waiting for a prompted clip to changing actions and inspecting a scene’s response.
Within the 30-second original, top and bottom start from the same image. Around 4–8 seconds, the lower left-hand tool crosses the foreground while the upper panel retains its higher grasp. At 16 seconds, the upper tissue has lifted and the lower continuation differs. The 15-second original also compares two paths, but starts from another scene. These are visible features of the supplied footage, not results reproduced with our own action inputs.
Both panels carry a generation label; neither should be treated as ground-truth surgical footage. The comparison illustrates a useful way to inspect action conditioning: hold the starting point fixed and vary the controls. Attractive tissue motion by itself does not establish command following.
Sources and further reading
- NVIDIA: from tabletop tasks to clinical scenesResearch-team article · October 20Model, distillation, VR-action comparisons and future research; clinical validation remains to be done.
- 60-second original demo: controllers and generated videoAuthors’ original video · no audio trackThe opening shows the operator’s console; later footage shows a labelled generated scene. This article uses the whole frame at 14 seconds. We inspected one-second samples across all 105 seconds of the three clips without running the model.
- 30-second original: two action paths from one sceneAuthors’ top/bottom comparison · no audio trackTop and bottom share an initial image, then take different tool paths. We retain whole frames at 0 and 16 seconds.
- 15-second original: two paths in another initial sceneAuthors’ top/bottom comparison · no audio trackDifferent action continuations in another scene; its initial image is not the same as the 30-second video.
From tabletop suturing to in-body scenes: inspect the training scope
The predecessor, Cosmos-H-Dreams, studied dVRK tabletop suturing. The new model builds on Cosmos3-H-Embodied-Nano in the Cosmos 3 family, then specializes on Versius cholecystectomy video and synchronized motion data. Rather than explicitly specifying each organ’s geometry, contacts and deformation equations, it learns scene continuation from paired video and actions.
The 28 action dimensions cover both arms’ poses, rotation and jaws, plus energy and thumbstick controls. Training selection requires exactly two visible, engaged tools and excludes active camera commands. The card reports 64 training recordings and about 526 unique minutes. The 4,792-recording source pool is not the selected training set, and this scope does not establish support for every procedure, instrument or camera movement.
This explains why looking surgical and being physically correct require separate evidence. The imagery can show occlusion, reflections and traction without measuring contact force, tissue stiffness or anatomical error. The authors list coupling with numerical simulation, surgeon evaluation and full-procedure coverage as future work. Visual realism is a reason to begin validation.
Sources and further reading
- NVIDIA: from tabletop tasks to clinical scenesResearch-team article · October 20Model, distillation, VR-action comparisons and future research; clinical validation remains to be done.
- Model card: training scope, outputs and author performanceNVIDIA model card · pinned revisionTraining selection, 28-dimensional actions, hardware and resolutions; not a BitShovel model test.
43.2 FPS does not measure controller-to-eye latency
The team first trains a bidirectional teacher, distills it into a causal student and uses Self-Forcing to train on the student’s own generated context. Further two-step training produces a separate checkpoint after the four-step student. It is not simply a four-step model with its sampling count changed to two.
On RTX PRO 6000 Blackwell Workstation Edition at 832×480, the authors report 30.0 FPS for four steps and 43.2 FPS for two. The DGX Spark result of 16.9 FPS uses 512×288 and two steps. Different hardware, resolutions and checkpoints cannot be collapsed into one speed ranking or assumed to deliver equal quality. We did not rerun these measurements.
Training actions are aligned at 10 Hz. A live system also has input acquisition, generation, queues, networking and headset display. The runtime guide says 30 Hz acquisition can be retimed into a 10 Hz model timeline, and raising display FPS does not increase generation throughput. One pose-to-JPEG metric even excludes networking and headset rendering. A real-time claim still leaves end-to-end latency and long-rollout consistency to inspect.
Sources and further reading
- NVIDIA: from tabletop tasks to clinical scenesResearch-team article · October 20Model, distillation, VR-action comparisons and future research; clinical validation remains to be done.
- Model card: training scope, outputs and author performanceNVIDIA model card · pinned revisionTraining selection, 28-dimensional actions, hardware and resolutions; not a BitShovel model test.
- Runtime guide: checkpoints, presets and three clocksOriginal repository guide · pinned commitTwo- and four-step weights use different presets; model, acquisition and display time are separate.
- The two separate public checkpointsPinned model directory · exp15Two- and four-step safetensors whose filenames specify bf16 are about 30.35GB each, with separate SHA-256 files; we did not download or run the weights.
What matters next is interaction that can be inspected repeatedly
For robotics researchers, such an environment could let a policy try different actions from a fixed scene, inspect how failures emerge and decide what deserves physical testing. What is publicly available is a research simulator. Policy evaluation, synthetic data and autonomy are intended uses or research directions, rather than demonstrated patient outcomes.
To try the system, match the two- or four-step checkpoint to its documented preset, then check memory and control requirements. The default configuration exceeds 32GB VRAM; live control also needs two tracked controllers and HTTPS. An ordinary computer cannot simply inherit the authors’ results. The public model files match the package guide, while the top-level README retains a previous-version notice. Pin the revisions and verify loading rather than relying on the model name.
The most useful next evidence would preserve the initial frame, actions, seed, weights and cache settings for others; show long trajectories, reversals and failures; and check structure and tissue response using measurements and expert evaluation. This article separates visible differences in the supplied footage from author-reported throughput. The work opens a direction for interactive generated environments, with reliability still to be examined through those concrete questions.
- Read the pinned runtime and timing guide
- Continue to MicroDuck simulation and physical transfer
- Open this model’s version and evaluation entries
Sources and further reading
- NVIDIA: from tabletop tasks to clinical scenesResearch-team article · October 20Model, distillation, VR-action comparisons and future research; clinical validation remains to be done.
- Model card: training scope, outputs and author performanceNVIDIA model card · pinned revisionTraining selection, 28-dimensional actions, hardware and resolutions; not a BitShovel model test.
- Runtime guide: checkpoints, presets and three clocksOriginal repository guide · pinned commitTwo- and four-step weights use different presets; model, acquisition and display time are separate.
- Requirements and live-control entryRepository README · pinned commitLinux, NVIDIA GPU, over 32GB VRAM by default and two-controller WebXR; model assets are supplied separately.
- The two separate public checkpointsPinned model directory · exp15Two- and four-step safetensors whose filenames specify bf16 are about 30.35GB each, with separate SHA-256 files; we did not download or run the weights.