Cosmos

Website | Framework

Introduction

NVIDIA Cosmos is an open platform of world models, datasets, and tools that enables developers to build Physical AI for robots, autonomous vehicles, smart infrastructure, and more.

Cosmos 3

Cosmos 3 is our newest model family [Report] [Website]. It is a suite of omnimodal world models designed to jointly process and generate language, images, video, audio, and action sequences within a unified Mixture-of-Transformers architecture. By supporting highly flexible input-output configurations, it seamlessly unifies critical modalities for Physical AI — effectively subsuming vision-language models, video generators, world simulators, and world-action models into a single framework.

Cosmos 3 exposes two runtime surfaces:

Key Capabilities

• World understanding: Analyze videos and images for captions, temporal events, next actions, spatial grounding, physical plausibility, and causal outcomes.
• World generation: Produce images, videos, synchronized sound, and action-conditioned rollouts from text, image, video, or action inputs.
• Action modeling: Predict policy actions, inverse dynamics, and forward dynamics for robotics, camera motion, egocentric motion, and autonomous-driving settings.
• Research and production paths: Use Diffusers and Transformers for Python-first development, then vLLM-Omni and vLLM for OpenAI-compatible serving.
• Post-training recipes: Adapt vision, action, and reasoner workflows with Cosmos Framework training recipes and task-specific evaluation [Coming Soon].

Model Architecture

Cosmos 3 is an omnimodal world model built on a unified Mixture-of-Transformers (MoT) architecture that combines an autoregressive (AR) transformer for reasoning with a diffusion transformer (DM) for multimodal generation. In Reasoner Mode, language and visual understanding tokens are processed through causal self-attention, enabling next-token prediction for tasks such as perception, planning, and world reasoning. In Generator Mode, noisy image, video, audio, and action tokens are denoised through full attention, allowing the model to jointly generate coherent multimodal outputs. Both modes share the same transformer architecture, multimodal attention layers, and a unified 3D multi-dimensional rotary position embedding (mRoPE) representation that encodes spatial and temporal structure across modalities, enabling consistent reasoning over images, videos, audio streams, and action trajectories.

Model Family

Supported Generation Settings

Input and Output

Use Cases

Generator

Generator examples produce non-text outputs conditioned by text, vision, and action inputs.

Generator prompt upsampling expands short scene descriptions into dense structured prompts. The current examples use these sampling defaults:

Reasoner

Reasoner examples produce text outputs from text and vision inputs. It follows Qwen3-VL-compatible message conventions for image and video inputs.

Reasoner examples use the following sampling settings:

Use this basic message shape for text + vision requests:

[
  {
    "role": "system",
    "content": [{"type": "text", "text": "You are a helpful assistant."}]
  },
  {
    "role": "user",
    "content": [
      {"type": "video_url", "video_url": "https://example.com/video.mp4"},
      {"type": "text", "text": "List the notable events with approximate timestamps."}
    ]
  }
]
copy

For explicit reasoning, append this format instruction to the user prompt:

Answer the question using the following format:

**\<think>**
Your reasoning.
**\</think>**

Write your final answer immediately after the </think> tag.
copy

Quickstart

Before running examples, create a Hugging Face access token and then authenticate locally:

uvx hf@latest auth login
copy

Set HF_HOME if you want to use a shared cache or a disk with more space.

Generator with Diffusers

uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
uv pip install --torch-backend=auto \
  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
  accelerate \
  av \
  cosmos_guardrail \
  huggingface_hub \
  imageio \
  imageio-ffmpeg \
  torch \
  torchvision \
  transformers
copy

import torch
from diffusers import Cosmos3OmniPipeline
from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
from diffusers.utils import export_to_video

pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Nano",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0)

result = pipe(
    prompt="A mobile robot navigates a warehouse aisle and stops at a shelf.",
    negative_prompt="",
    image=None,
    num_frames=189,
    height=720,
    width=1280,
    fps=24,
    num_inference_steps=35,
    guidance_scale=6.0,
    enable_sound=False,
    add_resolution_template=False,
    add_duration_template=False,
    generator=torch.Generator(device="cuda").manual_seed(1234),
)

export_to_video(result.video, "cosmos3_t2v.mp4", fps=24, macro_block_size=1)
copy

Generator with vLLM-Omni

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$(pwd):/workspace" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano \
  --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --allowed-local-media-path / \
  --port 8000 \
  --init-timeout 1800
copy

uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
# CUDA 13 driver:
uv pip install --torch-backend=cu130 \
  "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@refs/pull/3454/head"
# CUDA 12.8 driver:
# uv pip install --torch-backend=cu128 \
#   "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@refs/pull/3454/head"
copy

curl -sS -X POST http://localhost:8000/v1/videos/sync \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string "negative_prompt=blurry, distorted, low quality" \
  --form-string "size=1280x720" \
  --form-string "num_frames=189" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=6.0" \
  --form-string "flow_shift=10.0" \
  --form-string "seed=0" \
  --form-string 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -o cosmos3_t2v_output.mp4
copy

curl -sS -X POST http://localhost:8000/v1/videos/sync \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string 'extra_params={"guardrails":false,"use_resolution_template":false,"use_duration_template":false}' \
  -o cosmos3_t2v.mp4
copy

# no_guardrails.yaml
async_chunk: false
stages:
  - stage_id: 0
    max_num_seqs: 1
    enforce_eager: true
    trust_remote_code: true
    model_class_name: Cosmos3OmniDiffusersPipeline
    model_config:
      guardrails: false
      offload_guardrail_models: false
copy

Reasoner with Transformers

Coming soon!

Reasoner with vLLM

Use vLLM for Reasoner production inference behind an OpenAI-compatible chat-completions API.

uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
uv pip install --torch-backend=cu130 "vllm==0.21.0" \
  "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3"
copy

The vLLM version and the torch backend are paired: use --torch-backend=cu130 "vllm==0.21.0" for a CUDA 13 driver, or --torch-backend=cu128 "vllm==0.19.1" for CUDA 12.8. vLLM does not publish wheels for every CUDA minor version, so --torch-backend=auto is not reliable here — pick the pair that matches your driver.

vllm serve nvidia/Cosmos3-Nano \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --async-scheduling \
  --allowed-local-media-path / \
  --port 8000
copy

For notebook launch commands (Cosmos3-Super on four GPUs, media-path defaults, and full flag sets), see cookbooks/cosmos3/README.md — Start the server.

If your vLLM build reports that DeepGEMM is unavailable, disable it before starting the server:

export VLLM_USE_DEEP_GEMM=0
copy

Configuration notes:

Troubleshooting

Which CUDA version should I use?

CUDA 13 (recommended) or 12.8. Your system CUDA and PyTorch's CUDA major version must match — check with nvidia-smi and python -c "import torch; print(torch.version.cuda)".

Which base container should I use?

NVIDIA NGC PyTorch: nvcr.io/nvidia/pytorch:25.09-py3 for CUDA 13, or nvcr.io/nvidia/pytorch:25.06-py3 for CUDA 12.

torch.cuda.is_available() is False ("The NVIDIA driver on your system is too old")

The installed torch is newer CUDA than your driver — uv pip install torch defaults to CUDA 13 (cu130). Install a matching build: uv pip install --torch-backend=auto torch torchvision (or pin, e.g. --torch-backend=cu128). For uv sync notebooks use COSMOS3_UV_GROUP=cu128-train; for vLLM pair cu128 with vllm==0.19.1.

Import fails with libxcb.so.1: cannot open shared object file

On headless servers and minimal containers, importing or running a pipeline can fail with libxcb.so.1: cannot open shared object file (or another missing graphics library), because a dependency links against system X11/graphics libraries that are not installed. Install them:

apt-get install -y libxcb1 libgl1 libglib2.0-0
copy

uv errors on install or sync

The Cosmos Framework requires uv >= 0.11.3 (enforced via its pyproject.toml). Older versions fail to parse the project config (for example the [tool.uv.audit] section) and do not recognize newer --torch-backend values such as cu130 (you may see a value is required for '--torch-backend' or an accepted-values list that stops at cu129). Upgrade with uv self update (or reinstall from https://astral.sh/uv).

Choosing an Integration

Examples

We are building examples that show Cosmos 3 capabilities end to end, including world generation, world understanding, captioning, temporal localization, grounding, and physical reasoning. Each example is a self-contained script or notebook you can run from this repository.

Example	Surface	Workflows demonstrated	Open	nbviewer
Generator (audiovisual) with Diffusers	Generator	Text-to-image, plus text-to-video and image-to-video each with or without synchronized sound, via Cosmos3OmniPipeline.	Notebook
Generator (audiovisual) with Cosmos Framework	Generator	Text-to-image, plus text-to-video and image-to-video each with sound on or off, through the cosmos_framework.scripts.inference entrypoint.	Notebook
Generator (audiovisual) with vLLM-Omni	Generator	Text-to-image, plus text-to-video and image-to-video each with sound on or off, against an OpenAI-compatible vLLM-Omni server.	Notebook
Forward dynamics with Cosmos Framework	Generator	Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, through the cosmos_framework.scripts.inference entrypoint.	Notebook
Forward dynamics with vLLM-Omni	Generator	Forward dynamics: action-conditioned future-observation prediction for AV, DROID, and UMI, against an OpenAI-compatible vLLM-Omni server.	Notebook
Inverse dynamics with Cosmos Framework	Generator	Inverse dynamics: ego-motion trajectory prediction from input AV video, through the cosmos_framework.scripts.inference entrypoint.	Notebook
Inverse dynamics with vLLM-Omni	Generator	Inverse dynamics: ego-motion trajectory prediction from input AV video, against an OpenAI-compatible vLLM-Omni server.	Notebook
Reasoner with Cosmos Framework	Reasoner	Text and image reasoning: detailed captioning, robot task planning, 2D grounding, describe-anything, and action-trajectory prompts, through the cosmos_framework.scripts.inference entrypoint.	Notebook
Reasoner with vLLM	Reasoner	Image and video reasoning: captioning, temporal localization, embodied reasoning, common-sense reasoning, 2D grounding, describe-anything, action CoT, driving scenes, physical-plausibility, and situation understanding, against an OpenAI-compatible vLLM server (Cosmos3-Super on 4 GPUs by default; switch to Nano per the cookbook README).	Notebook

Inference Benchmarks

Cosmos 3 latency and serving numbers live in inference_benchmarks.md. Generator sections report diffusion-path latency (seconds) by GPU, engine, resolution, and tensor-parallel width; the Reasoner section reports vLLM serving metrics under concurrent load. Empty cells mean a combination has not been measured yet, not that it is unsupported.

Limitations

Cosmos 3 can produce artifacts in long, high-resolution, or physically complex outputs. Common failure modes include temporal inconsistency, unstable camera or object motion, inaccurate sound-video alignment, imperfect action-state consistency, object morphing, inaccurate 3D structure, and implausible physical dynamics. Applications that require physically grounded simulation, safety-critical control, or complex multi-agent behavior need additional validation, guardrails, and system-level safety analysis before deployment.

Ecosystem

News

• [May 31, 2026] We released Cosmos 3 in the NVIDIA Cosmos 3 Hugging Face collection and Cosmos Framework which provides runnable setup, inference, training, and evaluation workflows for Cosmos models. See the Cosmos 3 Technical Report.

License and Contact

This project may download and install additional third-party open source software projects. Review the license terms of those projects before use.

NVIDIA Cosmos source code and models are released under, and subject to the terms of, the OpenMDW-1.1 License. For a custom license, contact cosmos-license@nvidia.com.