WorldSimProbe

Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation

Peterson Co^1,2,3,*,†, Sicheng Hu^1,2,*, Chunxuan Jiao^1,2,*, Hongyang Cheng², Yulin Luo¹, Yijie Xu^2,4 Sixiang Chen¹, Zhongxia Zhao², Zihao Wang⁵, DaFeng Chi³, Peidong Liu³, YuTong Chen^2,6, Henghua Liu^2,6 Zhihao Yuan³, Huizhu Jia¹, Yuzheng Zhuang³, Tianle Zhang³, Liang Lin³, Huajie Tan^2,†, Shanghang Zhang^1,✉

¹State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, ²EvoPhys AI, ³Joy Future Academy, JD ⁴The University of Sydney, ⁵The Hong Kong University of Science and Technology, ⁶Beijing Institute of Technology

^* Equal contribution. ^† Project leaders. Corresponding author: shanghang@pku.edu.cn

GitHub arXiv Hugging Face PDF

Evaluation gap

A rollout can look right for the wrong reason.

Most ACWM evaluations focus on visual quality, task success, or coarse responsiveness, leaving two gaps: action realization is treated as a monolithic capability rather than a multi-dimensional process spanning local, global, and source-specific control variations, while interaction response is rarely checked for grounding in realized motion, valid contact, and correct dynamics. Yet a faithful simulator must preserve the full chain from supplied action to agent motion to environment response; otherwise, a rollout can look plausible or succeed while still ignoring controls or hallucinating unsupported interactions.

There exists a distribution gap between what simulators can do and what ACWMs are tested on (task distribution).

Failure modes

Where the causal chain breaks.

Action Calibration Failure

Action calibration is the alignment between the magnitude of an action change and its physical effect. Miscalibration shifts or obscures the failure threshold—the smallest perturbation beyond which task failure or rollout outcome change persists.

One standard. One chain.

A faithful simulator must preserve both links.

WorldSimProbe defines the Observable Simulator Contract, a single standard for simulator faithfulness: supplied actions must produce corresponding agent motion, and environment responses must be grounded in that motion. It traces this contract along one chain—from action realization to interaction response—through five controlled diagnostic suites.

r̂ Realized motion agent future

contact gate

r̂ ≈ Φ_R(r, e, a) ê ≈ Φ_E(e, r̂)

Interactive probes

Test the causal chain yourself.

Simulator Waiting

Connect to begin

Arm Left Command -- Steps 0

LingBot-VA Ready

Waiting for an action The selected model will run with the same trace.

Model ready --

Translate

Rotate

Operational overview

Five probes reveal where the chain fails.

Five diagnostic suites

Inspect each failure in aligned rollouts.

Action realization Interaction response

Local Action Calibration

Action Effects Are Coarsely Represented. Models capture broad action changes but fail to distinguish their effects with simulator-level precision. This coarse response scaling can delay the predicted onset of task failure and semantic outcome changes as perturbations increase.

Simulator GT

LingBot-VA

Benchmark results

One score cannot capture every failure.

LingBot-VA · RoboTwin

Action realization Interaction response

Benchmark scores by model and diagnostic suite
Model	T1	T2	T3	T4	T5

We’ve open-sourced the evaluation code and sample data. Run it on your model.

Citation

@misc{co2026worldsimprobe,
  title        = {{WorldSimProbe}: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation},
  author       = {Co, Peterson and Hu, Sicheng and Jiao, Chunxuan and Cheng, Hongyang and Luo, Yulin and Xu, Yijie and
                  Chen, Sixiang and Zhao, Zhongxia and Wang, Zihao and Chi, DaFeng and Liu, Peidong and Chen, YuTong and Liu, Henghua and
                  Yuan, Zhihao and Jia, Huizhu and Zhuang, Yuzheng and Zhang, Tianle and Lin, Liang and Tan, Huajie and Zhang, Shanghang},
  year         = {2026},
  eprint       = {2608.09298},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url          = {https://arxiv.org/abs/2608.09298}
}

WorldSimProbe

A rollout can look right for the wrong reason.

Where the causal chain breaks.

Action Calibration Failure

Global Trajectory Coverage Failure

Source Behavior Preservation Failure

Interaction Grounding Failure

Interaction Dynamics Failure

A faithful simulator must preserve both links.

Test the causal chain yourself.

Five probes reveal where the chain fails.

Inspect each failure in aligned rollouts.

Local Action Calibration

Global Trajectory Coverage

Action-Source Behavior Preservation

Interaction Grounding

Interaction Dynamics

One score cannot capture every failure.

Citation