Object-centric Cross-modal Feature Distillation for Event-based Object Detection

Event cameras are gaining popularity due to their unique properties, such as their low latency and high dynamic range. One task where these benefits can be crucial is real-time object detection. However, RGB detectors still outperform event-based detectors due to the sparsity of the event data and missing visual details. In this paper, we propose a cross-modality feature distillation method that can focus on regions where the knowledge distillation works best to shrink the detection performance gap between these two modalities. We achieve this by using an object-centric slot attention mechanism that can iteratively decouple feature maps into object-centric features and corresponding pixel-features used for distillation. We evaluate our novel distillation approach on a synthetic and a real event dataset with aligned grayscale images as a teacher modality. We show that object-centric distillation allows to significantly improve the performance of the event-based student object detector, nearly halving the performance gap with respect to the teacher.

Authors

• Lei Li
• Alexander Liniger
• Mario Millhäusler
• Vagia Tsiminaki
• Yuanyou Li
• Dengxin Dai

Keywords

• Visualization
• Image coding
• Detectors
• Object detection
• Gray-scale
• Feature extraction
• Real-time systems
• Feature Distillation
• Distillation For Object Detection
• Event Data
• Feature Maps
• Detection Performance
• Grayscale Images
• Performance Gap
• High Dynamic Range
• Distillation Method
• Event Dataset
• Dynamic Vision Sensor
• Real-time Object Detection
• Convolutional Neural Network
• Bounding Box
• RGB Images
• Real-world Datasets
• Graph Neural Networks
• Gated Recurrent Unit
• Student Model
• Foreground Objects
• Foreground Regions
• Event Frames
• Feature Alignment
• Affinity Matrix
• Attention Map
• Auxiliary Task
• Attention Matrix
• Two-stage Detectors
• Feature Pyramid Network