We introduce SimWorld, a new simulator built on Unreal Engine 5, designed for developing and evaluating LLM/VLM agents in rich, real-world-like settings.

🌍 Realistic World Simulation

Accurate physical and social dynamics with language-driven procedural environment generation

🤖 Rich Agent Interface

Multi-modal world inputs/feedback and open-vocabulary action outputs at varying abstraction levels

🎯 Diverse Scenarios

Physical and social reasoning scenarios that are easily customizable by users

Why SimWorld?

Large language models excel in structured domains but struggle in real-world conditions. SimWorld bridges this gap with unprecedented realism and flexibility.

Current Limitations

❌ Game-like environments lack realism
❌ Domain-specific simulators are too narrow
❌ Limited natural language interfaces
❌ Focus only on indoor scenes

SimWorld Solutions

✅ Photorealistic Unreal Engine 5 graphics
✅ Open-ended, procedural world generation
✅ Full language-driven control
✅ City-scale environments

Simulator	Open-ended / Procedural	Open-ended / Lang-Ctrl	Physical / Social Realism	Action / Abstraction	Action / Open-Vocab	Agent Type	Physics Engine
Minedojo	✅	❌	⭐️	Low-level	❌	Humanoid	Minecraft
Mindcraft	✅	❌	⭐️	High-level	❌	Humanoid	Minecraft
MetaUrban	✅	❌	⭐️⭐️	Low-level	❌	Vehicle	PyBullet
EmbodiedCity	❌	❌	⭐️⭐️⭐️	Low-level	❌	Drone/Vehicle	Unreal Engine
CARLA	❌	❌	⭐️⭐️⭐️	Low-level	❌	Vehicle	UE / Unity
GRUtopia	❌	❌	⭐️⭐️	Low-level	❌	Humanoid/Robot	Isaac Sim
OmniGibson	❌	❌	⭐️⭐️	High-/Low-level	❌	Robot	Omniverse
AI2-THOR	✅	❌	⭐️⭐️	Low-level	❌	Robot	Unity
Habitat 3.0	❌	❌	⭐️⭐️	Low-level	❌	Humanoid/Robot	Bullet
SimWorld	✅	✅	⭐️⭐️⭐️	High-/Low-level	✅	Humanoid/Robot/Vehicle	Unreal Engine

🌍 Infinite World Generation

SimWorld generates infinite worlds with diverse shapes, scales, and distributions using procedural generation. Users can easily control and edit the world layout through both code and a user-friendly UI.

🏙️ Various and Dynamic Simulated Scenes

Control environmental factors on the fly - change weather, time of day, and object properties during runtime using the Python API.

🚦 Advanced Traffic System

Dynamic traffic system simulates real-world traffic flow and pedestrian behavior with both rule-based and LLM-driven control mechanisms.

🤖 Various Agent Actions

Support for navigation, social behavior, object manipulation, and environment interaction through Python API.

👥 Social Reasoning Simulation

Define social reasoning tasks, generate corresponding scenarios, and evaluate agent capabilities in social contexts.

⚙️ Physical Reasoning Simulation

Embodied reasoning with multimodal perception and real-time decision-making under uncertainties.

📚 Citation

@misc{simworld,
      title={SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds},
      author={Zhuang*, Yan and Ren*, Jiawei and Ye*, Xiaokang and He, Xuhong and Gao, Zijun and Wu, Ryan and Dogra, Mrinaal and Zhang, Cassie and Kim, Kai and Wolfinger, Bertt and Ma, Ziqiao and Shu$^{\dagger}$, Tianmin and Hu$^{\dagger}$, Zhiting and Qin$^{\dagger}$, Lianhui},
      booktitle={Preprint(Under Review)},
      year={2025}
  }

Copied to clipboard!

SimWorld

An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds

NeurIPS 2025 Spotlight