Multi-PrefDrive: Optimizing Large Language Models for Autonomous Driving Through Multi-Preference Tuning

This paper introduces Multi-PrefDrive, a framework that significantly enhances LLM-based autonomous driving through multidimensional preference tuning. Aligning LLMs with human driving preferences is crucial yet challenging, as driving scenarios involve complex decisions where multiple incorrect actions can correspond to a single correct choice. Traditional binary preference tuning fails to capture this complexity. Our approach pairs each chosen action with multiple rejected alternatives, better reflecting real-world driving decisions. By implementing the Plackett-Luce preference model, we enable nuanced ranking of actions across the spectrum of possible errors. Experiments in the CARLA simulator demonstrate that our algorithm achieves an 11. 0% improvement in overall score and an 83. 6% reduction in infrastructure collisions, while showing perfect compliance with traffic signals in certain environments. Comparative analysis against DPO and its variants reveals that Multi-PrefDrive’s superior discrimination between chosen and rejected actions, which achieving a margin value of 25, and such ability has been directly translates to enhanced driving performance. We implement memory-efficient techniques including LoRA and 4-bit quantization to enable deployment on consumer-grade hardware and will open-source our training code and multi-rejected dataset to advance research in LLM-based autonomous driving systems. Project Page (https://liyun0607.github.io/).

Authors

• Yun Li
• Ehsan Javanmardi
• Simon Thompson
• Kai Katsumata
• Alex Orsholits
• Manabu Tsukada

Keywords

• Training
• Codes
• Translation
• Quantization (signal)
• Urban areas
• Hardware
• Complexity theory
• Safety
• Autonomous vehicles
• Tuning
• Language Model
• Large Language Models
• Light Signal
• Preferred Model
• Pedestrian
• System Architecture
• Reference Model
• RGB Images
• Optimal Policy
• Reward Function
• Dataset Construction
• Human Preferences
• Traffic Scenarios
• Traffic Rules
• Preference Ranking
• Multimodal Input
• Reward Model