arXiv Notify me• Jun 26, 2026paper

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Jim Fan(guest)

Get the full transcript next time arXiv releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on Jim Fan.

Automating Real-to-Sim for Scalable Robotics Training Bridging the Sim-to-Real Gap with High-Fidelity Simulation

0 quotes

Concerns Raised

The computational cost and scalability of the automated scene generation pipeline are not detailed.
The system's performance on highly cluttered or dynamic scenes beyond the tested tasks remains an unknown.
Potential edge cases or failure modes in the zero-shot sim-to-real transfer are not discussed.

Opportunities Identified

Accelerating the development of generalist robots by providing a scalable engine for training data generation.
Establishing new standardized benchmarks for policy evaluation that have high predictive power for real-world performance.
Enabling the creation of vast, diverse datasets required for training large-scale robot foundation models.

Key Themes

Research Findings7

Robot policies trained on data generated by SimFoundry can be transferred zero-shot to real-world tasks, including multi-step manipulation, articulated object interaction, and bimanual interaction.

In an evaluation across 7 manipulation tasks and 5 policy architectures, SimFoundry's simulation performance strongly predicted real-world performance, showing a mean Pearson correlation of 0.911 and a mean maximum ranking violation of 0.018.

Robot policies trained in SimFoundry with "object cousins" (object variations) showed an average task success rate improvement of 17% when transferred zero-shot to the real world.

Robot policies trained in SimFoundry with "scene cousins" (scene variations) showed an average task success rate improvement of 21% when transferred zero-shot to the real world.

Robot policies trained in SimFoundry with "task cousins" (task variations) showed an average task success rate improvement of 40% when transferred zero-shot to the real world.

SimFoundry is a modular and automated system for zero-shot real-to-sim scene construction from a video.

SimFoundry generates sim-ready digital twins and supports object, scene, and task editing to create "digital cousins," which are affordance-preserving variations of real-world scenes.

Topics

Robotics Simulation Sim-to-Real Transfer Digital Twins Policy Learning Machine Learning Artificial Intelligence Data Augmentation Scene Generation Robot Manipulation Zero-Shot Learning NVIDIA Research Computer Vision Generalization Bimanual Manipulation

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

Automating Real-to-Sim for Scalable Robotics Training

Bridging the Sim-to-Real Gap with High-Fidelity Simulation

Enhancing Generalization with 'Digital Cousins'

Enabling Zero-Shot Transfer for Complex Manipulation

Research Findings7

Topics