Just Released: The World’s First Event-Level Prediction Embodied AI World Model!

< img id="wx_img" src="https://www.qbitai.com/wp-content/uploads/imgs/qbitai-logo-1.png" width="400" height="400">

Just Now, the World’s First “Event-Level Prediction” Embodied AI World Model Has Arrived!

2026-05-29 15:02:05 Source: QbitAI

From frame-by-frame action learning to event-based world understanding

By Meng Yao, from Aofei Temple

QbitAI | WeChat Official Account QbitAI

Ask a robot to hand you a cup—

This seemingly simple task is, for current embodied large models, an exam requiring逐帧 (frame-by-frame) completion:

Predict where the hand will be in 0.1 seconds, then in 0.2 seconds…

Slice a complete action into dozens of nearly identical frames, forcing the model to learn one frame at a time.

The result? The model memorizes “fingers move X millimeters per frame,” not the goal “grasp the cup.” Change the cup, change the table, or slightly alter the timing—and it instantly fails!!

Just now, Ziliang Robotics has unveiled a novel solution—

Releasing WALL-WM, the world’s first “event-level prediction” embodied AI world model.

WALL-WM shifts the prediction unit of world modeling from temporal frames to semantic events:

Instead of asking “what will it look like in 0.1 seconds?”, the model directly imagines what the scene looks like *at the moment the cup is grasped*, skipping all redundant intermediate frames, and simultaneously generates the trajectory needed to reach that state.

Since “events” themselves are cross-scenario, cross-object semantic abstractions, WALL-WM demonstrates significantly stronger generalization across scenarios. This model has been published in the paper 《WALL-WM: Carving World Action Modeling at the Event Joints》.

Ah, now it’s settled.

From now on, small robots can work more like humans—focusing on key points and flexibly handling various dramatic situations in the physical world!

From Frame-Based Action Learning to Event-Based World Understanding

Over the past few years, mainstream VLA models have largely followed one path:

Feed the model a single current frame plus a language instruction, and ask it to predict a fixed-length block of future actions.

This approach is certainly engineering-friendly and convenient for training—but the problem is, real-world robot actions do *not* obediently occur within fixed time windows.

For instance, asking a robot to pick up a cup involves at least several stages: approaching, contact, closing the gripper, lifting, moving, and placing. Each stage exhibits distinct physical states; pre-contact and post-contact dynamics represent entirely different control problems.

Addressing this flaw, Ziliang Robotics presents a highly “counterintuitive” industry insight in their paper:

Text, vision, and action are inherently impossible to “fully align”… (doge)

The paper notes that text, vision, and action reside in high-dimensional spaces with distinct “manifold geometries” and vastly different “time scales.”

Text encodes high-level, low-entropy semantic intent; vision provides continuous, high-dimensional observations; action is strongly constrained by physics—extremely sensitive to contact states, timing precision, and minor perturbations.

If these three modalities are directly compressed into a shared space, pre-trained representations easily deviate from their original geometric priors!!

That’s precisely why many VLAs in the industry perform far worse on real robots in visual-language-action alignment than their underlying VLMs would suggest…

Given these persistent issues with traditional VLAs, the Ziliang team re-examined a more fundamental question: *At what granularity should a robot learn an action?*

Based on this, they developed the WALL-WM world model, enabling robots to train and execute in an *event-centric* manner.

Simply put, “event-centric” means segmenting robot tasks at genuinely semantically meaningful and physically transformative *event boundaries*, then training the model on these event-labeled data.

Examples include reaching, grasping, lifting, relocating, and placing—each constitutes a semantically coherent action-centered event.

Such events can be clearly described in language, fully captured in video, and mapped onto robot trajectories—thus truly linking language, vision, and action.

Here lies the key to WALL-WM’s superior generalization: letting robots understand world changes through events, then translating that understanding into executable actions.

And *this* is the true form a embodied AI “world model” should take.

WALL-WM’s Core Pipeline: Previsualize First, Then Execute

Specifically, WALL-WM does *not* directly generate actions from input frames.

Instead, it first enables the model to understand *how the world will change due to the next event*, then translates that change into the robot’s required trajectory.

Underpinning this is a full reconstruction of the perception-to-control pipeline, which Ziliang divides into three layers:

Layer 1: Event Instruction Input Its role is straightforward—to tell the model *what to do next*, e.g., “pick up the cup,” “place it in the basket,” or “arrange the block at the designated position.”

Layer 2: Event-Centric World Model Given the specified event, the model previsualizes how the scene will evolve: how objects move, how the environment changes, and how the robotic arm should participate.

Layer 3: Multi-View Spatiotemporal Fusion Robots typically observe from multiple angles—e.g., head-mounted and wrist-mounted cameras provide complementary perspectives. WALL-WM unifies these views, ensuring the model comprehends the full scene before executing actions.

Moreover, within this architecture, WALL-WM incorporates several key designs to transform this pipeline into a system that preserves video priors while developing robust action capabilities.

One Backbone, Two Inference Modes

During execution, WALL-WM does *not* generate rigid, fixed-length action sequences. Instead, the same model weights support two distinct inference modes:

First is the Event Mode. When a higher-level planner has already decomposed the task, the model directly outputs a variable-length action sequence based on the event description. This mode embodies WALL-WM’s core idea: actions need not be forcibly segmented into fixed windows but should naturally unfold along semantic event boundaries.

The second is the Unified Mode. When no external planner exists—and the robot must perceive, reason, and act simultaneously—the VLM integrates current visual input and task instructions to generate intermediate reasoning online, then passes the result to the action model to output fixed-length action blocks.

This mode suits real-time closed-loop control better, as it maintains stable control frequency.

Crucially, both modes share the same set of weights, and can even switch between action blocks during execution—eliminating the need to retrain for different scenarios. Thus, the model is highly flexible: it can either serve as a dedicated executor for pre-planned events within larger robotic systems, or autonomously handle the full pipeline—from understanding tasks and deciding next steps to generating actions.

Separated Growth of Video and Action Models

Furthermore, WALL-WM does *not* directly convert video models into action models. Instead, it decouples and grows these two capabilities separately:

Let the robot first previsualize how the world will change, then decide how to move.

Concretely, the video model inherits dynamic priors from internet-scale video training, focusing on understanding object motion and scene evolution.

The action model, initialized from scratch, specializes in translating these visual changes into robot trajectories.

They couple unidirectionally at each layer: the action stream reads visual evidence from the video stream, while the video stream retains its original dynamic priors—preventing premature distortion by action data.

Thus, the model preserves the world-understanding capability of its video backbone while allowing action capabilities to scale robustly during large-scale training—a feat most VLAs fail to achieve.

Geometric-Aware Multi-View Fusion

As everyone knows, real-world robots typically employ multiple cameras—for example, an overhead view for global context and a wrist camera for fine-grained hand details.

However, multi-view inputs do *not* naturally align. Simple cross-view attention often leads the model to learn feature mixing—correlating regions that appear related but violate actual spatial geometry.

To solve this, WALL-WM introduces two mechanisms:

1. Frustum Masking:

Using camera calibration data, it determines whether two image patches could physically observe the same 3D region; geometrically inconsistent associations are pruned from the attention pathway. This ensures cross-view attention respects real-world geometry.

1. Tube Masking:

It randomly masks a contiguous spatiotemporal region in one view, preventing the model from relying solely on intra-view temporal cues and forcing it to seek clues from other cameras.

One mechanism eliminates erroneous connections; the other enforces cross-view dependency. Combined with calibration-free, learnable camera pose encodings, this naturally supports large-scale mixed training across multiple bodies and viewpoints.

Thus, cross-view attention evolves from an optional capability into a repeatedly exercised geometric correspondence skill during training.

Staircase Chain-of-Thought Decoding

In real physical environments, robots often need to “think” before executing complex tasks.

Chain-of-Thought (CoT) improves such decision quality—but traditional token-by-token generation is too slow. While slight delays are tolerable for chat models, robots cannot afford latency in action control.

To address this, WALL-WM proposes Staircase Layer-Relay CoT Decoding, preserving interpretable reasoning chains while optimizing decoding efficiency:

It transforms the original serial, token-by-token process into a structure where the *lower layer runs once*, and *higher layers expand in parallel*.

Specifically, the bottom layer extracts shared reasoning states in a single pass; subsequent reasoning tokens are generated in parallel at higher layers.

The output remains a continuous CoT latent sequence, which can be reconstructed into textual reasoning traces via a frozen LLM—preserving interpretability while reducing token-by-token decoding latency.

Thus, for the first time, interpretability and real-time performance no longer require trade-offs.

Behind the Event-Level World Model: A System-Level Reconstruction from Data to Deployment

WALL-WM aims to solve far more than just event-level architectural modifications.

What truly powers this capability is a comprehensive *system engineering* pipeline—from data collection and hierarchical annotation to sampling and training.

In data structure, WALL-WM does *not* rely solely on real-robot data. Instead, it constructs a Data Pyramid:

• Base layer: Millions of general web videos, enriching open-world visual and motion priors;
• Middle layers: Human action videos, first-person videos, public robot datasets, and self-collected video-action pairs;
• Top layer: Real-robot teleoperation, correction, and recovery data.

Each layer represents a controlled relaxation of constraints from the layer below—higher layers closer to real deployment, lower layers closer to open-world visual priors.

Moreover, to truly integrate events into training, WALL-WM avoids feeding raw robot trajectories as monolithic videos.

Instead, it adopts four-level hierarchical annotation + dual clustering sampling, decomposing each trajectory into Task → Subtask → Action → Segment layers. Thus, the model processes clearly bounded behavioral units—not混杂 (mixed) long sequences.

A notable finding in the paper: when text descriptions are segmented along action boundaries, both language distribution and vision-language joint distributions become significantly more balanced.

This implies that rare instructions and special scenario combinations—previously drowned out in long tasks—are naturally exposed to the model during training.

This approach not only helps the model grasp action boundaries but also improves data distribution, making long-tail samples easier to learn.

Beyond model and data, WALL-WM adds a dedicated low-level training system:

Current event-level modeling must jointly handle video, action, multi-view inputs, and long sequences—making training extremely costly. Without robust infrastructure, even brilliant methods cannot scale.

Ziliang’s solution: adopt a distributed “Muon” framework to enhance convergence and stability (DMuon), and use *multi-event packing*—embedding multiple events into a single long sequence—to reduce per-sample computational waste.

At deployment, knowledge distillation reduces denoising steps, and FP8 quantization cuts memory and inference costs—bringing this large model closer to the latency requirements of real-time robot control.

Experimental Results

In concrete experiments, WALL-WM’s value manifests immediately in large-scale *real-robot generalization*.

It not only executes fixed-template tasks but also supports event-centric textual inputs at varying granularities—and continues to perform action reasoning and execution under new instructions, objects, scenes, tasks, and robot embodiments.

• Embodied Video Generation: Outperforms Wan2.1/Wan2.2 across all three embodied dimensions—Motion Quality, Semantic Consistency, and Physical Plausibility:

• 3D Awareness (CO3Dv2): Surpasses WAN2.1-14B, Open-Sora 2.0, V-JEPA, and DINOv2 in Point Error and Depth Error:

• Real-Robot Core15 L1 Benchmark: Achieves significantly higher task completion scores than π0.5 and DreamZero across basic tasks, reasoning tasks, dexterous manipulation, and generalization scenarios—and ranks among the highest-performing L1 models under abstract instruction settings:

At the paper’s outset, the Ziliang Robotics team quotes a line from Plato’s *Phaedrus*:

“Follow the natural order, and let things take their course.”

When viewed in the context of the entire embodied intelligence industry, this statement is thought-provoking and highlights the core of WALL-WM.

Real-world physical tasks never occur neatly within fixed time windows; instead, they resemble a series of naturally connected events: reaching, touching, grasping, moving, and placing. Each critical change corresponds to a natural joint in the action.

What WALL-WM does is enable the model to understand the world, predict changes, and generate actions along these "event joints."

This also provides a more natural pivot point for the robot's generalization capabilities:

When language, objects, scenes, task combinations, or even the robot's own body change, it can still determine where it is in the process, how the world will change next, and how the action should be executed based on event boundaries.

Currently, competition in the embodied intelligence industry is shifting from benchmarking and demo showcases to real-world deployment. The industry focus is also moving from who appears more capable of movement to "who better understands changes, organizes actions, and achieves stable generalization."

This time, Self-Variable Robotics has already presented the leading results of this path with a coherent engineering paradigm.

Reference links:

[1] GitHub: https://github.com/X-Square-Robot/wall-x

[2] Project homepage:

https://x2robot.com/pages/wm

_All rights reserved. No reproduction or use in any form without permission. Legal action will be taken against violators._