A video model learns a distribution over sequences of frames. The Sora report describes some regularities of the physical world in generated videos, including camera motion consistent in 3D and objects persisting while hidden from view. The report argues that scaling video generation is a promising path toward physical-world simulators (Brooks et al., 2024).
Concept 97 of 100Video World ModelsGenerative Models · Study phase 13: Recent research (2024–2025)
Question
What is the difference between generating a video that looks right and predicting what would happen after an action, and which of the two does planning need?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
A video model learns a distribution over sequences of frames. The Sora report describes some regularities of the physical world in generated videos, including camera motion consistent in 3D and objects persisting while hidden from view. The report argues that scaling video generation is a promising path toward physical-world simulators (Brooks et al., 2024).
Action-conditioned video models can act as interactive environments: Genie, an 11-billion-parameter model trained without any action labels, lets a user act in generated scenes frame by frame (Bruce et al., 2024).
The same report lists failures, such as not modeling the physics of glass shattering, and controlled experiments find generalization by similarity to training cases rather than by physical rules (Brooks et al., 2024; Kang et al., 2025). A video that looks plausible is not evidence of a correct model of physics.
Questions to Think About
How could you test whether a video model has learned a physical law rather than reproducing similar training clips (Kang et al., 2025)?
Genie infers actions from videos that come without action labels; what can such a learned latent action represent, and what can it miss?
Core Math
Key Equation
pθ(x1:T∣c)=t=1∏Tpθ(xt∣x<t,c)
A video model learns a distribution over sequences of frames x1:T, possibly conditioned on text or other context c. It can generate frames one after another,
pθ(x1:T∣c)=t=1∏Tpθ(xt∣x<t,c),
or be a diffusion model that denoises a whole clip, often in a compressed latent space z:
θminEz,t,ϵϵ−ϵθ(zt,t,c)2
OpenAI describes Sora as a diffusion model of this second kind, a transformer over patches of compressed video (Brooks et al., 2024).
A world model in the reinforcement-learning sense must also respond to actions: it predicts pθ(xt+1∣x≤t,at), so that an agent can ask what would happen if it acted. Genie learns a small set of latent actions from unlabelled internet videos, which lets a user control generated environments frame by frame (Bruce et al., 2024).
Whether generating plausible video requires learning physical laws is an open question. Kang et al. (2025) trained video models on simple simulated physics and found that they generalized well within the training distribution but failed outside it, imitating the most similar training example instead of applying a general rule.