MMCD: Multi-Modal Collaborative Decision-Making for Connected Autonomy with Knowledge Distillation

Autonomous systems have advanced significantly, but challenges persist in accident-prone environments where robust decision-making is crucial. A single vehicle’s limited sensor range and obstructed views increase the likelihood of accidents. Multi-vehicle connected systems and multi-modal approaches, leveraging RGB images and LiDAR point clouds, have emerged as promising solutions. However, existing methods often assume the availability of all data modalities and connected vehicles during both training and testing, which is impractical due to potential sensor failures or missing connected vehicles. To address these challenges, we introduce a novel framework MMCD (Multi-Modal Collaborative Decision-making) for connected autonomy. Our framework fuses multi-modal observations from ego and collaborative vehicles to enhance decision-making under challenging conditions. To ensure robust performance when certain data modalities are unavailable during testing, we propose an approach based on cross-modal knowledge distillation with a teacher-student model structure. The teacher model is trained with multiple data modalities, while the student model is designed to operate effectively with reduced modalities. In experiments on connected autonomous driving with ground vehicles and aerial-ground vehicles collaboration, our method improves driving safety by up to 20. 7%, surpassing the best-existing baseline in detecting potential accidents and making safe driving decisions. More information can be found on our website https://ruiiu.github.io/mmcd.

Authors

• Rui Liu 0040
• Zikang Wang
• Peng Gao 0007
• Yu Shen
• Pratap Tokekar
• Ming Lin 0003

Keywords

• Training
• Connected vehicles
• Uncertainty
• Decision making
• Collaboration
• Robot sensing systems
• Data models
• Safety
• Testing
• Accidents
• Collaborative Decision-making
• Multiple Modalities
• Teacher Model
• Point Cloud
• Autonomous Vehicles
• RGB Images
• Data Modalities
• Student Model
• Ground Vehicles
• LiDAR Point Clouds
• Sensor Failure
• Truth Labels
• Data Sharing
• Decision-making Model
• Binary Cross-entropy Loss
• Attention Weights
• Communication Range
• Lidar Data
• RGB Camera
• RGB Data
• Collaborative Framework
• Final Embedding
• Collaborative Setting
• Soft Targets
• LiDAR Sensor
• Decision-making Capabilities
• Distillation Loss
• Left Turn