Monocular Real-to-Sim Scene Programming

Programming Interactive Scenes from Monocular Images for Embodied Simulation

NeurIPS 2026

Yumeng He^*, Yichen Song^*, Xiaotian Yang, Weijia Zhang, Zanwei Zhou,
Junru Gong, Xiaokang Yang, Yunbo Wang^†

Shanghai Jiao Tong University

* Equal contribution† Corresponding author

Arxiv Code

Abstract

The advancement of Embodied AI requires high-quality simulation assets that faithfully mirror the physical world. NeoWorld-Pro reformulates monocular scene reconstruction as a procedural programming task for interactive 3D environments. It leverages multimodal large language models to translate a single RGB image into executable programs defining object geometry, articulation, physical properties, and scene layout.

To ensure simulation readiness, NeoWorld-Pro introduces a physics-in-the-loop mechanism that executes generated programs in a physics engine and iteratively refines them using simulation feedback. The resulting scenes support stable stacking, fine-grained manipulation, and articulated object interactions that are difficult for open-loop reconstruction pipelines.

Interactive Scene

Select an input image to explore its interactive simulated scene.

Simulated Interactive Scene

Choose an input image to switch the simulation.

Articulated Asset

Inspect generated assets and actuate their articulated parts in the browser.

USD Preview

Loading demo

Preparing the interactive viewer.

Loading 3D scene...

Choose a case to load its simulation-ready asset.

Image-to-policy demonstration

Programming simulation-ready scenes for embodied manipulation.

Policy Rollout

Choose an input image to switch the demonstration.

Dataset and Benchmark

A multi-object simulation benchmark for physically executable scenes.

NeoWorld-Pro is evaluated on PartNet-Mobility and a newly constructed synthetic scene benchmark designed to stress-test monocular reconstruction in physically realistic multi-object environments. The benchmark tasks cover placement, assembly, articulation, and interaction with task-relevant affordances.

Across 84 downstream manipulation tasks, NeoWorld-Pro achieves a 92.85% task success rate.

90 articulated categories

30 USD-format scenes

84 downstream tasks

The first three groups of NeoWorld-Pro benchmark scenes with their corresponding manipulation tasks.

Citation

BibTeX

@inproceedings{he2026neoworld,
  title={{NeoWorld-Pro}: Programming Interactive Scenes from Monocular Images for Embodied Simulation},
  author={He, Yumeng and Song, Yichen and Yang, Xiaotian and Zhang, Weijia and Zhou, Zanwei and Gong, Junru and Yang, Xiaokang and Wang, Yunbo},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026}
}