Acting and Imagining Together

Humans imagine what happens next while they act.

What Is a World Model?

An actor chooses an action.

A world model imagines what happens next.

A critic evaluates the imagined outcome.

DreamerV3, MuZero, TD-MPC2

• [Hafner et al., 'Mastering Diverse Control Tasks through World Models']
• [Schrittwieser et al., 'Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model']
• [Hansen et al., 'TD-MPC2: Scalable, Robust World Models for Continuous Control']

JEPA-Style Latent World Models

Learn to predict future representations without game-reward supervision.

• [Zhou et al., 'DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning']
• [Assran et al., 'V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning']
• [Maes et al., 'LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels']

The Data behind Latent World Models

Current observation, numerical action, future observation.

The Usual Numerical-Action Predictor

Visual tokens and embedded numerical actions condition a Transformer.

The action interface and training data define the supported domains.

Actions Differ across Domains

Vector size and physical meaning both change.

What We Mean by Generative World Models

Video generation models, such as Cosmos-Predict2, generate future frames from the current frame and a text prompt.

• [Ali et al., 'World Simulation with Video Foundation Models for Physical AI']

Language Describes Visual Change

Video generation models learn from diverse images and videos.

• [Nan et al., 'OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-Video Generation']
• [Wei et al., 'OmniEdit: Building Image Editing Generalist Models through Specialist Supervision']

A Broad Visual Prior

Language makes heterogeneous visual data usable in a shared prediction task.

Video Pretraining Can Support Policy Learning

Cosmos-Predict2 predicts future video.

Cosmos Policy learns actions and their visual consequences.

• [Ali et al., 'World Simulation with Video Foundation Models for Physical AI']
• [Kim et al., 'Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning']

Scaling Training Data for Latent World Models

A shared prediction task can absorb increasingly diverse visual data.

LaLa-World Architecture

Encode observations into visual representations.

A language model reads the condition alongside visual tokens.

• [Qwen Team, 'Qwen3.5: Towards Native Multimodal Agents']

Planning toward a Goal Image

Given an initial state and a goal, find an action sequence.

Sampling Candidate Sequences

Imagining Candidate Futures

Latent Goal Comparison

MPC Executes, Observes, and Replans

Draw action sequences and express each one in language.

Roll out the latent world model for each sequence.

Rank candidates by the L2 distance in latent space.

Keep the best candidates, refit, and sample again.

• [Rubinstein, 'Optimization of Computer Simulation Models with Rare Events']

Planning across Five Shared Benchmarks

LaLa-World achieves the highest average success rate.

• [Terver et al., 'What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?']
• [Zhou et al., 'DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning']
• [Maes et al., 'LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels']

Selected Actions in Wall

Numerical language actions

• [Zhou et al., 'DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning']

Selected Actions in Push-T

Numerical language actions

• [Chi et al., 'Diffusion Policy: Visuomotor Policy Learning via Action Diffusion']

Selected Actions in PointMaze

Qualitative language actions

• [Fu et al., 'D4RL: Datasets for Deep Data-Driven Reinforcement Learning']

Selected Actions in Rope

Qualitative language actions

• [Zhang et al., 'AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation']

How the Policy Recipes Differ

VLAs predict actions, but not their consequences

• [Physical Intelligence et al., '${\pi }_{0.5}$: a Vision-Language-Action Model with Open-World Generalization']
• [Li et al., 'CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification']

How the Policy Recipes Differ

The initialization and prediction targets distinguish these representative models.

• [Physical Intelligence et al., '${\pi }_{0.5}$: a Vision-Language-Action Model with Open-World Generalization']
• [Li et al., 'CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification']
• [Kim et al., 'Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning']
• [Ali et al., 'World Simulation with Video Foundation Models for Physical AI']

World-Model Pretraining Transfers to Policy Learning

LaLa-Policy is competitive with strong VLA and video-policy baselines.

• [Physical Intelligence et al., '${\pi }_{0.5}$: a Vision-Language-Action Model with Open-World Generalization']
• [Fang et al., 'MolmoAct2: Action Reasoning Models for Real-World Deployment']
• [Kim et al., 'Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning']
• [Nasiriany et al., 'RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots']

Faster Policy Inference

LaLa-Policy predicts future latents while keeping action performance competitive.

• [Kim et al., 'Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning']
• [Nasiriany et al., 'RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots']

Acting and Anticipating Together

The policy jointly predicts an action chunk and a future latent.

Acting and Anticipating Together

The video compares execution with a prediction made earlier.

LIBERO: Bowl to Plate

“Pick up the black bowl on the ramekin and place it on the plate.”

• [Liu et al., 'LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning']

LIBERO: Bowl to Cabinet

“Put the bowl on top of the cabinet.”

• [Liu et al., 'LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning']

RoboCasa: Serve Coffee

“Press the button on the coffee machine to serve coffee.”

• [Nasiriany et al., 'RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots']

RoboCasa: Open the Microwave

“Open the microwave door.”

• [Nasiriany et al., 'RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots']

Extending the Current Work

Test whether the gains transfer to real robots.

Robotics Grows with Agents

My hypothesis: robotics can grow as rapidly as AI agents.

Research Questions at the Agent–Robot Interface

Where can a latent world model be most useful?