Anyone who has watched an AI-generated video has probably seen it quietly break: a dog runs behind a couch and its collar vanishes, or the camera pans back and the loveseat has turned into a sofa. These glitches are a symptom of how many of today’s models work — and a growing body of research on “world models” aims to fix them.
Why AI video drifts
Part of the problem is that many AI systems are fundamentally predictive. The models behind chatbots like ChatGPT are trained to predict the next piece of text; video-generation models predict what is statistically most plausible to appear next on screen. In neither case does the system maintain an explicit, continuously updated model of the scene it is depicting. Without that internal map, details drift because nothing enforces consistency from one frame to the next.
Researchers across several AI fields are now working to build such internal representations, and the implications reach well beyond video generation into robotics, autonomous vehicles and the broader pursuit of more capable AI.
From 3D illusions to 4D models
One useful frame is the idea of a four-dimensional (4D) model — three spatial dimensions plus time. Consider the 2012 stereoscopic 3D re-release of Titanic. Freezing a frame conveys a sense of depth between characters and objects, but the depth is an illusion built from stereoscopy: two images, one per eye, shown in rapid alternation. Everyone in the cinema sees the same fixed pair of perspectives; no one can “walk around” an actor to see his face from another angle.
Advances over the past decade have made genuine perspective changes increasingly feasible. Beginning around 2020, the Neural Radiance Field (NeRF) approach showed how photorealistic new viewpoints could be synthesised from a set of images, effectively reconstructing a scene so it can be viewed from angles never actually photographed.
Extending this idea to moving footage yields something closer to 4D: a scene that can be navigated through both time and space. A recent preprint, “NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos”, describes reconstructing ordinary videos into 4D models in order to generate new views along novel camera trajectories.
Using 4D models to stabilise generation
The same technology can feed back into video creation. A related preprint, “TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Model”, applies directly to the opening example: its authors argue that a continuously updated 4D world model, guiding the generation process, improves the stability of AI video. Such a model would help keep the loveseat from morphing into a sofa and the dog from losing its collar.
These are early results, but they point to a broader trend: models that maintain and update an internal scene map as they generate are more reliable and retain a form of short-term spatial memory. That memory also enables occlusion — digital objects correctly disappearing behind real ones. As a 2023 paper put the requirement, achieving occlusion requires a 3D model of the physical environment.
Beyond graphics: robots, vehicles and benchmarks
Rapidly converting video into 4D also produces rich training data for robots and autonomous vehicles learning how the physical world behaves. By building 4D models of their surroundings, robots can navigate more effectively and anticipate what might happen next. Current general-purpose vision-language models, which interpret images and text but do not build explicit world models, still stumble here: a benchmark paper presented at a 2025 conference reported “surprising limitations” in their basic world-modelling abilities, including near-random accuracy when reasoning about motion trajectories.
What “world model” means to those chasing AGI
The term carries a heavier meaning for researchers pursuing artificial general intelligence (AGI). Leading large language models already hold an implicit understanding of the world absorbed from their training data. Angjoo Kanazawa, an assistant professor of electrical engineering and computer science at the University of California, Berkeley, has noted that in a sense these models already possess a fairly good world model — the difficulty is that it is not well understood how they do it.
The catch, in her account, is that these implicit models are not a real-time, physical understanding of the world, because LLMs cannot update their training in real time. OpenAI’s own GPT-4 technical report notes that, once deployed, the model does not “learn from experience.” The open challenge, as Kanazawa frames it, is building a vision system that can take streaming input, continuously update its understanding of the world and act accordingly — a problem she regards as central to any path toward AGI.
Even researchers who doubt LLMs alone can reach AGI often see them as one component of future systems. In Kanazawa’s framing, an LLM could serve as a language-and-common-sense interface, while a more explicit underlying world model supplies the spatial and temporal memory current models lack.
A high-profile bet on world models
That thesis now has prominent backers. In late 2025, Yann LeCun announced he was leaving Meta to launch a startup — now Advanced Machine Intelligence (AMI Labs) — to build systems that understand the physical world, hold persistent memory, and can reason and plan complex sequences of tasks. He had laid the groundwork in a 2022 position paper that asked why humans cope well in unfamiliar situations, arguing the answer may lie in learning internal “world models” of how the world works.
Evidence for the value of internal models continues to accumulate. An April 2025 Nature paper reported results on DreamerV3, an AI agent that improves its behaviour by learning a world model and “imagining” future scenarios — the first agent to obtain diamonds in Minecraft from scratch without human data. Commercial efforts are emerging too, such as World Labs’ Marble world model.
So while “world model” in the AGI sense refers to an internal model of how reality works rather than a 4D reconstruction, progress in 4D modelling can supply components that help with vision, memory and short-term prediction. Along the way, 4D models can also provide rich simulations of reality in which to test AI systems — so that when they are deployed, they are better prepared to operate in the real world. Related shifts in multimodal AI are explored in this overview of unified multimodal models.
Limitations and what to watch
Much of this work is early-stage. The NeoVerse and TeleWorld results are recent preprints that, at the time of writing, reflect their authors’ own reported findings rather than long-term independent validation. There is also no single agreed definition of a “world model”: the term spans everything from a 4D scene reconstruction to an abstract internal model of physical reality, and progress in one sense does not automatically translate to the other. Whether explicit world models are truly necessary for AGI, or whether scaled-up existing approaches can close the gap, remains an open and actively debated question. The near-term, verifiable gains are concrete but narrow — steadier video generation, better training data for robotics — while the largest claims about general intelligence stay speculative.
