Revive driving scene simulation by simulator-conditioned generative models
Yunsong Zhou, Michael Simon, Zhenghao Peng, Sicheng Mo, Hongzi Zhu, Minyi Guo, and Bolei Zhou
- Presented by MetaDriverse, GenForce, and Shanghai Jiao Tong University
- 📬 Primary contact: Yunsong Zhou ( [email protected] )
- arXiv paper | Blog TODO | Slides
🔥 The first simulator-conditioned generative model for controllable driving scene generation with appearance and layout diversity.
🌟 SimGen addresses simulation to reality (Sim2Real) gaps via cascade diffusion paradigm, and follows layout guidance from simulators and cues of the rich text prompts to realistic driving scenarios.
📊 DIVA dataset comprises 147.5 hours of web videos and synthesized data for diverse scene generation and advancing Sim2Real research.
[2024/06]SimGem paper released.[2024/06]DIVA dataset subset released.
- Release DIVA dataset
- Release SimGen code
- Toolkits for novel scene generation
DIVA-Real. It collects driving videos from YouTube, covering a worldwide range of geography, weather, scenes, and traffic elements and preserving the appearance diversity of a wide range of traffic participants. Here we provide a sample of 🔗 YouTube video list we used. For privacy considerations, we are temporarily keeping the complete data labels private.
DIVA-Sim. The Sim2Real data is induced from the same real-world scenarios, in which we can obtain real-world map topology, layout, and raw sensor data. It also includes hazardous driving behaviors through interactions introduced by adversarial traffic generation. The digital twins (on nuScenes dataset) and safety-critical scenarios (on Waymo Open dataset) can be obtained through this 🔗data link.
All assets and code in this repository are under the Apache 2.0 license unless specified otherwise. The annotation data is under CC BY-NC-SA 4.0. Other datasets (including nuScenes, Waymo, and MetaDrive) inherit their own distribution licenses. Please consider citing our paper and project if they help your research.
@article{zhou2024simgen,
title={SimGen: Simulator-conditioned Driving Scene Generation},
author={Zhou, Yunsong and Simon, Michael and Peng, Zhenghao and Mo, Sicheng and Zhu, Hongzi and Guo, Minyi and Zhou, Bolei},
journal={arXiv preprint arXiv:2406.09386},
year={2024}
}We acknowledge all the open-source contributors for the following projects to make this work possible:
You are welcome to follow other related work from 
- ELM | OpenScene | DriveAGI
- ScenarioNet | CAT | FreeControl