Robbyant, the embodied-AI unit within Ant Group, has open-sourced LingBot-World, a large-scale world model that turns video generation into an interactive simulator for embodied agents, autonomous driving, and games. Released under the Apache 2.0 license in January 2026, the system is designed to deliver controllable environments with high visual fidelity, strong dynamics, and long temporal horizons, while remaining responsive enough for real-time control. In plain terms: instead of producing a fixed video clip, the model produces a world that reacts — keyboard and mouse input change what happens next, the way they would in a game engine.
From text-to-video to text-to-world
Most text-to-video models generate short clips that look realistic but behave like passive movies; they do not model how actions change the environment over time. LingBot-World is designed as an action-conditioned world model. It learns the transition dynamics of a virtual world, so keyboard and mouse inputs, combined with camera motion, drive the evolution of future frames. Formally, the model learns a conditional distribution of future video tokens given previous frames, language cues, and individual actions. At training time it predicts sequences up to about 60 seconds; at inference time it can roll out coherent video streams extending to roughly 10 minutes while keeping scene structure stable.
The data engine: web video, game logs, and synthetic trajectories
A core design element is an integrated data engine that provides rich, aligned observations of how the world changes across realistic scenarios. The acquisition pipeline combines three sources: large-scale web videos of humans, animals, and vehicles from first- and third-person views; game data in which RGB frames are strictly coupled with user controls (such as W, A, S, D) and camera parameters; and synthetic trajectories rendered in Unreal Engine, where clean frames, camera intrinsics and extrinsics, and object layout are all known.
After collection, a profiling step standardizes this heterogeneous corpus: it filters for resolution and duration, splits videos into clips, and estimates missing camera parameters using geometry and pose models. A vision-language model scores clips for quality, motion magnitude, and scene type, then selects a curated subset. On top of this, a hierarchical captioning module creates three levels of text supervision — descriptive captions for complete trajectories including camera motion, static captions describing environment layout without motion, and dense temporal captions for short windows focused on local dynamics. This separation helps the model distinguish static structure from motion patterns, which matters for long-horizon stability.
Architecture: MoE video backbone and action conditioning
LingBot-World starts from Wan2.2, a 14B-parameter image-to-video diffusion transformer that already captures robust open-domain video priors. The Robbyant team expands this into a mixture-of-experts DiT with two experts of roughly 14B parameters each — 28B total — with only one expert active at each denoising step, keeping inference cost near that of the compact 14B model while expanding capacity. A training curriculum extends sequences from 5 seconds to 60 seconds, increasing the ratio of high-noise timesteps to stabilize global layout over long contexts and reduce mode collapse in long rollouts.
To make the model interactive, actions are injected directly into the transformer blocks. Camera rotation is encoded with Plücker embeddings; keyboard actions are represented as multi-hot vectors over keys such as W, A, S, D. These encodings are fused and passed through adaptive layer normalization, which modulates hidden states in the DiT. Only the action-adapter layers are fine-tuned — the main video backbone stays frozen — so the model retains pre-trained visual quality while learning action response from smaller interactive datasets. Training uses both image-to-video and video-to-video continuation, yielding a transition function that can start from arbitrary time points.
Distillation for real-time use
The intermediate model, LingBot-World Base, still relies on multi-step diffusion with full temporal attention, which is expensive for real-time interaction. The team therefore introduced LingBot-World Fast, an accelerated variant initialized from the high-noise expert, in which full temporal attention is replaced by block-causal attention — bidirectional within each temporal block, causal across blocks. This design supports key-value caching, letting the model stream frames autoregressively with low overhead.
Distillation uses a diffusion-forcing strategy: the student is trained on a small set of target timesteps, including timestep 0, so it sees both noisy and clean latents. Distribution-matching distillation is combined with an adversarial head, with the adversarial loss updating only the discriminator, which stabilizes training while preserving action adherence and temporal coherence. In the reported experiments, LingBot-World Fast reaches about 16 frames per second at 480p on a single GPU node and maintains end-to-end interaction latency under one second for real-time control.
Emergent memory and long-horizon behavior
One of the most notable reported behaviors is emergent memory: the model maintains global consistency without explicit 3D representations such as Gaussian splatting. When the camera moves away from a landmark and returns after about 60 seconds, the structure reappears with consistent geometry; when a car leaves the frame and later re-enters, it reappears in a physically plausible location rather than frozen or reset. The team also demonstrates coherent generation up to about 10 minutes with stable layout.
Benchmark results
For quantitative evaluation, the research team used VBench on a curated set of 100 generated videos, each longer than 30 seconds, comparing LingBot-World with two recent world models, Yume-1.5 and HY-World-1.5.
According to the reported scores, LingBot-World exceeds both baselines on imaging quality, aesthetic quality, and dynamic degree — with the largest margin on dynamic degree (0.8857 versus 0.7612 and 0.7217), indicating richer scene transitions and more complex motion responding to user input. Motion smoothness and temporal flicker are on par with the best baseline, and the model achieves the best overall stability metric among those compared. A separate comparison with interactive systems such as Matrix-Game-2.0, Mirage-2, and Genie-3 positions LingBot-World as one of the few fully open-source world models combining general domain coverage, long generation horizons, high dynamic degree, 720p resolution, and real-time capability.
Applications: worlds, agents, and 3D reconstruction
Beyond video synthesis, LingBot-World is positioned as a testbed for embodied AI. It supports prompted world events, where text instructions change weather, lighting, or style over time, or inject local events such as fireworks or moving animals while preserving spatial structure. It can also be used to train downstream action agents — for example, compact vision-language-action models such as Qwen3-VL-2B predicting control policies from images. Because generated streams are geometrically consistent, they can feed 3D reconstruction pipelines that produce static point clouds for indoor, outdoor, and synthetic scenes. This release sits alongside Ant Group’s broader embodied-AI efforts, including LingBot-VLA, a vision-language-action foundation model for robot manipulation.
Limitations and what to watch
- The benchmark figures and latency numbers are self-reported by the research team; independent replication on public checkpoints is still limited given the recency of the release.
- Real-time performance (about 16 FPS at 480p) was reported on a full GPU node — consumer hardware results will differ, and 480p remains below the fidelity of production game engines.
- “Emergent memory” holds over roughly minute-scale horizons; world models in general still struggle with strict physical consistency over long sessions, and generated physics should not be treated as a substitute for validated simulators in safety-critical training (for example, autonomous driving).
- The open-source world-model space is moving quickly, with closed systems such as Genie-3 and multiple open releases in 2025–2026; comparative standings may change rapidly.
Key takeaways
LingBot-World extends text-to-video into text-to-world simulation: keyboard actions and camera movement directly control long-horizon rollouts of up to about 10 minutes. It is trained on an integrated data engine combining web video, action-labeled game logs, and Unreal Engine trajectories with hierarchical captions that separate layout from motion. The backbone is a 28B-parameter mixture-of-experts diffusion transformer built from Wan2.2, with two 14B experts and action adapters fine-tuned over a frozen visual backbone. The distilled Fast variant achieves roughly 16 FPS at 480p on a single GPU node with sub-second latency. Full details are available in the official release announcement and the technical coverage at MarkTechPost.
