GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions

Paper
arXiv:2608.06332
Code
Coming soon
Overview of GeniWorld: an autoregressive robotic world model with visual action representations
Overview of GeniWorld. GeniWorld is an autoregressive robotic world model that transforms action inputs into visual action representations, enabling closed-loop interaction with human operators and robot policies. Trained on limited scene-specific demonstrations, GeniWorld generalizes to out-of-distribution (OOD) scenarios to produce high-fidelity observation predictions. Furthermore, it synthesizes rich manipulation trajectories to boost downstream policy performance and robustness under diverse conditions.
GeniWorld demo videos
We propose a generalizable interactive world model conditioned on embodied visual actions that enables closed-loop interaction with policies and human operators. Even with limited demonstrations, our model generalizes robustly across diverse scenarios and enables the generation of novel behaviors.

Abstract

Generalist robot policies exhibit strong capabilities, but their robustness in complex and unseen environments remains limited. Scaling robot learning and evaluation in diverse real-world environments remains costly and challenging. Action-conditioned world models offer a promising alternative, but they often suffer from limited action controllability and poor generalization to out-of-distribution (OOD) scenarios. To this end, we present GeniWorld, an interactive world model for robots that generalizes robustly across unseen scenarios. Building on pretrained video generative models, we use URDF-based rendering to transform numerical actions into visual action representations, enabling spatially grounded action control. By explicitly decoupling embodiment kinematics from environmental dynamics, our model mitigates scene overfitting and facilitates modeling of robot-environment interactions. To achieve closed-loop control, we construct an autoregressive video prediction model integrated with high-frequency robot kinematic control, enabling interaction with both robot policies and human teleoperators. In our experiments, even when trained solely on limited fixed-scene data, our model achieves superior in-domain performance and robust zero-shot generalization to highly randomized, unseen environments. For downstream applications, GeniWorld serves as a scalable policy evaluator that remains reliable under environmental perturbations. Furthermore, even with limited real-world demonstrations, GeniWorld generates diverse manipulation trajectories within the world model, improving downstream policy performance and robustness in complex environments.

Method

Overview of the GeniWorld method with URDF rendering and causal DiT
Overview of the GeniWorld method. We convert robot actions into visual motions via URDF rendering. These motions are subsequently encoded into latent representations and concatenated with noisy video latents. The combined representation serves as input to a causal DiT that predicts future videos via flow matching. During inference, the initial scene image is provided as the first frame. Actions are passed through the URDF-based renderer to produce motion conditions, and the model predicts future observations while leveraging KV caching to maintain historical context.

Results

Qualitative Clean-to-Random world-modeling results comparing GeniWorld with baselines
Qualitative Clean-to-Random world-modeling results. All models are trained on clean tabletop data and evaluated in unseen scenes. GeniWorld preserves the commanded robot motion and produces interaction outcomes that closely match the ground truth, whereas competing methods fail to generate the expected results in out-of-distribution (OOD) scenarios. Furthermore, the ablation results show that our dense visual actions enable more accurate interaction modeling.

Policy Improvement

Move Bowl

Only Real-World Data Fail Only Real-World Data · Distractor Distractor
Add GeniWorld-Synthesis Data Success Add GeniWorld-Synthesis Data · Distractor Distractor
Only Real-World Data Fail Only Real-World Data · Novel Instance Novel Instance
Add GeniWorld-Synthesis Data Success Add GeniWorld-Synthesis Data · Novel Instance Novel Instance

Place Mug

Only Real-World Data Fail Only Real-World Data · Distractor Distractor
Add GeniWorld-Synthesis Data Success Add GeniWorld-Synthesis Data · Distractor Distractor
Only Real-World Data Fail Only Real-World Data · Novel Instance Novel Instance
Add GeniWorld-Synthesis Data Success Add GeniWorld-Synthesis Data · Novel Instance Novel Instance

Open Drawer

Only Real-World Data Fail Only Real-World Data · OOD 1 OOD 1
Add GeniWorld-Synthesis Data Success Add GeniWorld-Synthesis Data · OOD 1 OOD 1
Only Real-World Data Fail Only Real-World Data · OOD 2 OOD 2
Add GeniWorld-Synthesis Data Success Add GeniWorld-Synthesis Data · OOD 2 OOD 2

Fold Towel

Only Real-World Data Fail Only Real-World Data · Lighting Lighting
Add GeniWorld-Synthesis Data Success Add GeniWorld-Synthesis Data · Lighting Lighting
Only Real-World Data Fail Only Real-World Data · Novel Instance Novel Instance
Add GeniWorld-Synthesis Data Success Add GeniWorld-Synthesis Data · Novel Instance Novel Instance
Real-world policy success rates across different GeniWorld data augmentation settings
Real-world policy success rates across different settings. Real denotes policies trained strictly on real-world data; Real + Spatial-Gen adds GeniWorld-synthesized data focused on spatial generalization; and Real + Spatial-Gen + Diverse-Gen adds both spatially randomized data and data with diverse scene variations. The combined regime achieves the strongest overall performance.

Citation

@article{gu2026geniworld,
  title={GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions},
  author={Gu, Chenghao and Yu, Hanyang and Zhang, Jingbo and Lin, Haitao and Zhang, Wenyao and Wang, Jinghe and Jin, Hanglei and Xie, Shuzhao and Jiang, Jingyan and Wang, Zhi},
  journal={arXiv preprint arXiv:2608.06332},
  year={2026}
}