World models W01

World models: objectives, data shapes, and training methods

A research map of what a world model is, why there is no single canonical architecture, and how its objective, data, representation, and training loop determine what it can actually do.

The short definition — a world model is a learned representation of how an environment evolves, especially how possible futures depend on the current state, the observed history, and an agent's actions. It is not necessarily a renderer, a policy, or a full simulator.

1 · The common core

In a partially observed environment, the model first compresses the history into an internal state or belief (z_t). It then predicts how that state changes under an action. Observation, reward, and continuation heads are optional:

$$z_t = E(o_{\le t}, a_{<t}), \qquad \hat z_{t+1} \sim P_\theta(z_{t+1}\mid z_t,a_t)$$ $$\hat o_{t+1}=D(\hat z_{t+1}), \qquad \hat r_t=R(z_t,a_t), \qquad \hat d_t=C(z_t,a_t)$$

Historyobservations, past actions, context

→

Encoder Eturns evidence into a belief state

→

Dynamics Prolls the state forward under candidate actions

→

Prediction headspixels, features, reward, value, risk

→

Useplanning, policy learning, simulation, representation

The narrow, agentic meaning requires action-conditioned counterfactuals: what happens if I do this? The broader generative meaning also includes action-free video predictors. Those can learn motion and regularities, but they do not become reliable intervention models until actions are observed, inferred as latent variables, or added during post-training.

How many variants exist? — there is no meaningful fixed count. “World model” names a combinatorial design space. At a useful coarse resolution there are four primary objective families, five common dataset shapes, and several independent representation, uncertainty, optimization, and deployment choices. Most named systems combine several cells of this map.

1 · What is predicted?pixels or tokens · latent features · structured physical state · reward/value/policy · mixtures

2 · Under which intervention?action-free · observed actions · inferred latent actions · goals or language instructions

3 · In which representation?raw observation · continuous latent · discrete token · objects/graphs/geometry · hybrid

4 · With which uncertainty?deterministic · stochastic latent · autoregressive distribution · diffusion/flow · ensemble

5 · From which data regime?offline trajectories · online interaction · passive video · self-play/simulation · multimodal mixtures

6 · How is it used?MPC · tree search · imagined actor–critic · synthetic data · frozen representation

2 · Four primary objective families

Objective	Target	What it buys	Main failure mode
Generative observation prediction	future pixels, sensor values, or visual tokens	inspectable rollouts and synthetic environments	capacity spent on nuisance detail; plausible video can still have wrong physics
Latent or feature prediction	future learned embeddings	efficient, semantic representations without decoding every pixel	the representation may discard information needed by a later task
Structured-state prediction	objects, poses, depth, occupancy, geometry, physical variables	interpretability, compositionality, and explicit constraints	requires supervision or architectural assumptions about the world
Task or control prediction	reward, value, policy, termination, risk, task-relevant latent states	learns only the distinctions needed for decisions	excellent for one utility function but incomplete for another
Hybrid	several targets above	balances fidelity, abstraction, and control	loss weighting and optimization interference become part of the research problem

A generative model often minimizes negative log-likelihood; a feature model predicts a stopped-gradient target embedding; a task-aware model predicts decision quantities:

$$\mathcal L_{\mathrm{gen}}=-\log p_\theta(o_{t+1:t+H}\mid o_{\le t},a_{t:t+H-1})$$ $$\mathcal L_{\mathrm{latent}}=d\!\left(F(z_t,a_t),\operatorname{sg}(E(o_{t+1}))\right)$$ $$\mathcal L_{\mathrm{task}}=\mathcal L_r+\mathcal L_v+\mathcal L_\pi$$

MuZero is the cleanest example of the third equation: it predicts reward, policy, and value without reconstructing observations. Dreamer-like systems are hybrid latent state-space models. V-JEPA predicts features rather than pixels. GAIA-1 and GameNGen emphasize high-fidelity generative futures.

3 · What the dataset actually looks like

The dataset's temporal structure determines which dynamics can be identified. A shuffled image collection contains appearance but no transition evidence; passive video contains transitions but not necessarily the causal effect of actions.

A · Individual transitions

(observation_t, action_t, reward_t, observation_t+1, terminal_t)

Enough when the observation is approximately Markovian. If relevant state is hidden, isolated transitions erase the history needed to infer it.

B · Fixed-length trajectory windows

observations  [B, T+1, C, H, W]
actions       [B, T, A]
rewards       [B, T, 1]
terminals     [B, T, 1]
mask          [B, T, 1]

This is the standard minibatch for image-based recurrent state-space models and temporal transformers. Vector observations replace C,H,W with D; multi-camera streams often add a view dimension V.

C · Complete episodes

Full trajectories preserve returns, long-range dependencies, rare failures, and goal completion. Training usually samples shorter windows from them while keeping episode boundaries and timestamps.

D · Passive video

video         [B, T, C, H, W]
actions       absent
rewards       absent

Abundant and useful for visual pretraining, but action effects are confounded with camera motion and the behavior of people or other agents. Genie learns a latent action space from video; V-JEPA 2 instead performs action-free pretraining and then adds action conditioning with a smaller robot dataset.

E · Multimodal embodied streams

RGB + depth + proprioception + force + audio
 language + maps + actions + reward

These streams may have different sampling rates and missing modalities. Alignment, timestamps, modality masks, and sensor calibration become part of the data model—not merely preprocessing.

Coverage beats volume — a huge observational dataset can contain almost no evidence about rare, dangerous, or deliberately unusual actions. A planner will query precisely these counterfactual regions and can exploit model errors there. Data collection policy is therefore part of world-model design.

4 · Training methodologies

Method	Mechanism	When it fits
One-step teacher forcing	predict (t+1) from the true history	stable baseline; weak match to long open-loop deployment
Multi-step rollout or overshooting	unroll learned dynamics for (H) steps and penalize drift	planning and long-horizon prediction; controls compounding error
Variational state-space training	learn a transition prior and an observation-conditioned posterior with prediction plus KL losses	partial observability and multimodal futures
Autoregressive token modeling	tokenize modalities and optimize next-token cross-entropy	large heterogeneous datasets and scalable generation
Diffusion or flow modeling	denoise a future observation or trajectory conditioned on history/actions	detailed, multimodal simulation; usually expensive inside a planner
Joint-embedding prediction	predict masked or future target features, without pixel reconstruction	representation learning and efficient latent planning
Online alternating learning	collect experience → update model → improve behavior in imagination → collect again	model-based reinforcement learning and active coverage
Offline pretrain + action post-train	learn visual regularities from broad video, then add controllability with scarce interaction data	robotics and physical-AI foundation models

Training and use should match. One-step accuracy is not evidence that a model remains coherent over a planning horizon. Likewise, a low video reconstruction loss is not evidence that its counterfactual actions are correct.

5 · Match the design to the goal

Objective: Masked or future latent-feature prediction
Data: Large passive-video corpus; labels optional
Training: Joint-embedding prediction with temporal masking
Use and evaluation: Frozen/fine-tuned encoder; probes, transfer, anticipation

Design selector. Choose the research goal: the useful world model changes with the question it must answer. These are starting configurations, not universal recipes.

6 · How a learned world model becomes useful

Model-predictive control (MPC): sample candidate action sequences, roll them out, execute the best first action, observe again, and replan.
Tree search: expand counterfactual action branches using learned reward, policy, and value predictions, as in MuZero.
Imagined actor–critic: train policy and value networks on latent rollouts, as in Dreamer.
Synthetic-data generation: create controllable scenarios for another perception or policy model.
Representation learning: freeze or fine-tune the encoder for downstream understanding.

7 · Evaluation and research traps

If the claim is…	Evaluate…	Do not rely only on…
accurate forecasting	held-out likelihood, calibrated uncertainty, error versus rollout horizon	one-step pixel similarity
causal or controllable dynamics	action interventions and counterfactual outcomes	passive-video plausibility
useful planning	real-environment success, return, safety, and sample efficiency	reconstruction metrics
general representation	transfer across tasks, scenes, embodiments, and goals	performance on the training utility alone
realistic simulation	temporal coherence, geometry, interaction physics, diversity, expert judgment	short cherry-picked clips

The central distinction is plausibility versus correctness. A generated future can look convincing while violating contact dynamics, geometry, or conservation laws. Prediction from observation is also not automatically causal: a useful intervention model needs action variation, assumptions, or an experimental data-collection process.

8 · Primary papers consulted

Consulted for this research map on July 16, 2026. Links point to the authors' primary papers rather than secondary summaries.

Year	Paper	Why it is in the map
2018	World Models — Ha & Schmidhuber	modular visual encoder, recurrent dynamics, and a controller trained in imagined rollouts
2019	Learning Latent Dynamics for Planning from Pixels (PlaNet) — Hafner et al.	deterministic-plus-stochastic latent state-space dynamics, overshooting, and latent MPC
2020	Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (MuZero) — Schrittwieser et al.	task-equivalent dynamics predicting reward, policy, and value without observation reconstruction
2023	Mastering Diverse Domains through World Models (DreamerV3) — Hafner et al.	hybrid recurrent state-space modeling and actor–critic learning from latent imagination
2024	TD-MPC2: Scalable, Robust World Models for Continuous Control — Hansen, Su & Wang	decoder-free implicit world modeling with latent trajectory optimization
2024	Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA) — Bardes et al.	self-supervised feature prediction without pixels, text, negatives, or pretrained encoders
2024	Genie: Generative Interactive Environments — Bruce et al.	unlabelled Internet video, discrete video tokens, autoregressive dynamics, and learned latent actions
2023	GAIA-1: A Generative World Model for Autonomous Driving — Hu et al.	multimodal video, text, and action token modeling for controllable driving scenarios
2025	Diffusion Models Are Real-Time Game Engines (GameNGen) — Valevski et al.	two-stage agent data collection and action-conditioned diffusion frame generation
2025	Cosmos World Foundation Model Platform for Physical AI — Agarwal et al.	video curation, tokenizers, pretrained foundation models, and downstream post-training
2025	V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning — Assran et al.	large action-free video pretraining followed by small action-conditioned robot post-training and MPC