Building interactiveworlds for embodiedintelligence.

In recent years, AI has advanced rapidly through a simple but powerful recipe: large-scale pretraining followed by reinforcement learning. Embodied intelligence, however, faces a much harder scaling problem. The physical world is continuous, dynamic, and expensive to interact with; robots must learn not only what they see, but also how objects move, respond to forces, and change through interaction. To bring embodied intelligence into the same scalable learning paradigm, we need environments where such experience can be generated at scale.

Science fiction has long imagined intelligence learning inside virtual worlds. In The Matrix, Neo acquires new abilities through interaction in a virtual environment. This idea may offer a promising direction for embodied intelligence.

Embodied intelligence needs worlds it can interact with, not just worlds it can see.

Real-to-sim-to-real provides a practical path toward this goal: reconstruct real environments in simulation, use them to generate large-scale data and interactive experience for pretraining and reinforcement learning, and transfer the resulting capabilities back to physical robots. However, building such environments remains difficult. Visual realism alone is not enough: reconstructed worlds must also support modification, interaction, articulation, and physical simulation. For embodied intelligence, reconstruction must therefore move beyond generating 3D assets toward building simulation-ready interactive worlds.

We introduce NeoWorld-3, an agentic framework for reconstructing simulation-ready interactive worlds from visual observations. While existing reconstruction systems have achieved impressive visual quality through direct prediction, reconstructing complex structures, interactions, and physical constraints remains challenging. The emergence of large vision-language models and coding agents creates a new opportunity to move beyond this paradigm.

NeoWorld-3 is designed around three core principles: simulation-ready reconstruction, coding-agent based world construction, and object-to-scene scaling. Simulation-ready reconstruction preserves the geometry, articulation, and physical validity required for downstream simulation. Coding-agent based reconstruction enables vision-language models to actively construct and refine worlds through tools. Object-to-scene reconstruction allows the same framework to scale from individual objects to complex environments.

Most 3D reconstruction methods generate a final representation directly from visual observations. NeoWorld-3 takes a different route: reconstruction is carried out by coding agents that write modeling programs, invoke tools, inspect intermediate results, and modify only the parts that need improvement. This is especially useful for scene reconstruction, where errors are often local—a wrong object geometry or pose should not require regenerating the entire scene.

FIGURE 01NEOWORLD-3 RECONSTRUCTION WORKFLOWswipe to pan →

THE RECONSTRUCTION ENGINE

Observe. Build. Refine.

Ⅱ Pause motion

01 SCENE-LEVEL RECONSTRUCTION

Scene-level feedback ↻

INPUT / MULTI-VIEW RGB-D

Real observations

Images, depth & camera poses

COORDINATOR

Scene agent

Global scene state

PlanDelegateRevise

SHARED TOOLKIT

NeoMCP + NeoSDK

SDK

Geometry constructionDifferentiable execution

MCP

Visual inspectionCode editing & rendering

cube()   translate()   rotate()

RECONSTRUCTED WORLD

Real-Sim scene

REAL

SIM

Object handoff

Views · source · context

02

OBJECT SUBAGENTLocal edits. Iterative feedback. Shared tools.

OBJECT-LEVEL RECONSTRUCTION

01 PROGRAM

Author & revise

object.py

def build(θ):
  body = shape(θ)
  add_parts(body)
  return object

Editable modeling programs

02 GEOMETRY

Refine geometry

θ₁ θ₂ θ₃

Optimize shape parameters

03 FEEDBACK

Compare & improve

REAL

SIM

Real-Sim visual feedback

04 ARTICULATION

Make it interactive

Add parts & joints

VALIDATION GATE

Scene integration

Validate object updates

✓ Accept↻ Revise

Object-level feedback ↻ Validated updates

03

DOWNSTREAM APPLICATIONSLearn in simulation. Transfer to the real world.

SIM-TO-REAL

Deploy to the real world

Optimized policyReal robot execution

LEARNING LOOP

Policy agent

TrainEvaluateImprove

SIMULATION

Policy optimization

Candidate policiesSimulation feedbackTransfer policy

Reconstruction flow Iterative feedbackHover a step to trace its connections

NeoWorld-3 organizes this process with a hierarchical reconstruction harness. A scene-level coordinator maintains the global scene and decomposes it into scoped object tasks. Each object is handled by a fresh worker with access to its relevant observations and editable state. NeoMCP exposes reconstruction operations to the agent, while NeoSDK provides the underlying geometry, rendering, and optimization primitives. Object reconstruction can therefore be developed and refined independently while remaining grounded in the shared scene.

Before an object update is merged back, the candidate is inspected against its visual evidence and scene context; unsuccessful changes can be revised or rejected. The updated scene then becomes the context for subsequent tasks. In practice, this lets NeoWorld-3 improve complex scenes incrementally, preserving successful reconstructions while concentrating computation and agent reasoning on the parts that remain incorrect.

One challenge in reconstruction is that vision-language models are strong at understanding visual concepts and structural relationships, but precise geometric parameter prediction remains difficult. Even when a model understands what an object should look like, directly predicting highly accurate geometry from numerical parameters is challenging.

NeoWorld-3 addresses this limitation through Differentiable Refinement. The agent first constructs an initial reconstruction, then refines it through differentiable rendering. The rendered result is compared with the original observation, image similarity serves as the optimization objective, and parameters in the generated geometric representation are updated through gradient-based optimization.

This creates a natural division of labor between reasoning and optimization. The vision-language model provides semantic understanding, decomposition strategies, and reconstruction programs, while differentiable optimization provides the numerical precision needed for high-fidelity geometry. Rather than relying on prediction alone, NeoWorld-3 can continuously improve reconstruction through visual feedback.

A visually accurate reconstruction is not necessarily a structurally valid one. Generated geometry can contain open boundaries, self-intersections, or inconsistent connections between parts, while movable components may lack explicit articulation structure. These issues are easy to overlook visually, but they prevent reconstructed objects from becoming reliable interactive assets.

NeoWorld-3 addresses these problems through explicit geometry validation and repair. The system inspects boundaries directly on the native indexed geometry, applies targeted repairs to invalid regions while preserving existing surface attributes, and re-inspects the repaired mesh to verify watertightness rather than relying on implicit remeshing. In parallel, the agent reconstructs articulation by identifying object parts, establishing joint relationships, estimating motion axes and ranges, and correcting invalid configurations.

This produces assets that are not only visually reconstructed, but also geometrically valid and explicitly articulated. The watertight pipeline preserves the original reconstruction while repairing structural defects, and the articulation pipeline captures how individual parts are connected and allowed to move. Together, they turn a reconstructed shape into a structured, interactive 3D asset.

Language models evolved from generating text to using tools and writing code. We believe 3D reconstruction is undergoing a similar transition. Coding-agent based reconstruction moves beyond directly predicting a final mesh: agents can reason about structure, construct representations, invoke tools, inspect intermediate results, and iteratively refine them.

NeoWorld-3 is a step toward this future. As foundation models improve, we aim to scale this approach from individual objects to complex real-world environments, turning visual observations into simulation-ready worlds for embodied learning and physical interaction. More broadly, we see reconstruction as a key layer in the embodied intelligence stack, connecting the physical world with scalable simulation. If this process can be made scalable, every real-world observation can become the starting point of a new training environment for intelligent agents.

Our long-term vision is to build the foundation for the next generation of embodied intelligence: A world where AI systems can not only perceive and understand, but also interact with and learn within it.