ReSim: Reliable World Simulation for Autonomous Driving

How can we reliably simulate future driving scenarios under a wide range of ego driving behaviors? Recent driving world models, developed exclusively on real-world driving data composed mainly of safe expert trajectories, struggle to follow hazardous or non-expert behaviors, which are rare in such data. This limitation restricts their applicability to tasks such as policy evaluation. In this work, we address this challenge by enriching real-world human demonstrations with diverse non-expert data collected from a driving simulator (e. g. , CARLA), and building a controllable world model trained on this heterogeneous corpus. Starting with a video generator featuring diffusion transformer architecture, we devise several strategies to effectively integrate conditioning signals and improve prediction controllability and fidelity. The resulting model, ReSim, enables Reliable Simulation of diverse open-world driving scenarios under various actions, including hazardous non-expert ones. To close the gap between high-fidelity simulation and applications that require reward signals to judge different actions, we introduce a Video2Reward module that estimates reward from ReSim’s simulated future. Our ReSim paradigm achieves up to 44% higher visual fidelity, improves controllability for both expert and non-expert actions by over 50%, and boosts planning and policy selection performance on NAVSIM by 2% and 25%, respectively.

Authors

• Jiazhi Yang
• Kashyap Chitta
• Shenyuan Gao The Hong Kong University of Science and Technology
• Long Chen Institute of Automation, Chinese Academy of Science Waytous
• Yuqian Shao
• Xiaosong Jia
• Hongyang Li Shanghai Artificial Intelligence Laboratory
• Andreas Geiger
• Xiangyu Yue The Chinese University of Hong Kong
• Li Chen BSH Home Appliances Holding (China) Co. , Ltd

Keywords

No keywords are indexed for this paper.