A world model predicts how an environment responds to an action, including the next state and reward. A learned model lets an agent improve its policy on simulated experience as well as real experience, which can reduce the number of real interactions it needs (Sutton & Barto, 2018, ch. 8).
Concept 76 of 100World ModelsTheory · Study phase 10: Mathematical foundations & information geometry
Question
Why can predicting the next latent state be easier than predicting the next frame pixel by pixel, and what can a model trained this way fail to represent?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
A world model predicts how an environment responds to an action, including the next state and reward. A learned model lets an agent improve its policy on simulated experience as well as real experience, which can reduce the number of real interactions it needs (Sutton & Barto, 2018, ch. 8).
Ha & Schmidhuber (2018) compressed game frames into a small latent space, learned the dynamics there, and showed that a controller could be trained entirely inside the model's simulated environment and then transferred to the real game.
DreamerV3 learns a latent world model and trains its policy on imagined rollouts; with one fixed set of hyperparameters it outperformed specialized methods on more than 150 tasks and was the first algorithm reported to collect diamonds in Minecraft from scratch, without human data or curricula (Hafner et al., 2025).
Questions to Think About
Why do small one-step prediction errors grow over long imagined rollouts, and why does keeping rollouts short and starting them from real states limit the damage (Janner et al., 2019)?
When is it cheaper to learn a model and plan with it than to learn a policy directly from real experience, and what happens when the model is wrong in exactly the states the policy prefers?
Core Math
Key Equation
s^t+1,r^t=fθ(st,at)
A world model is a learned model of how an environment responds to actions. Given the current state st and action at, it predicts the next state and the reward:
s^t+1,r^t=fθ(st,at)
In model-based reinforcement learning the agent uses this model to plan, or to train its policy on simulated experience, instead of learning only from real interaction. Dyna is the classic form: each real step is followed by several planning updates on transitions generated by the model (Sutton & Barto, 2018, §8.2).
Latent world models first compress each observation ot, such as a video frame, into a small latent vector and learn the dynamics there (Ha & Schmidhuber, 2018; Hafner et al., 2025):
Encoder: zt=enc(ot)
Dynamics: z^t+1=dyn(zt,at)
Decoder, used to train the latent: o^t=dec(zt)
A policy π(a∣z) can then be trained on imagined rollouts of H steps, maximizing the predicted return
Eπ,fθ[t=0∑H−1γtr^t].
The difficulty is compounding error: each imagined step feeds the model its own slightly wrong prediction, so long rollouts drift away from what the real environment would do. Janner et al. (2019) analyse this trade-off and use only short model rollouts that start from states the agent actually visited.
Key Papers and Books
Recurrent World Models Facilitate Policy Evolution