TrackOcc: Camera-Based 4D Panoptic Occupancy Tracking

Comprehensive and consistent dynamic scene understanding from camera input is essential for advanced autonomous systems. Traditional camera-based perception tasks like 3D object tracking and semantic occupancy prediction lack either spatial comprehensiveness or temporal consistency. In this work, we introduce a brand-new task, Camera-based 4D Panoptic Occupancy Tracking, which simultaneously addresses panoptic occupancy segmentation and object tracking from camera-only input. Furthermore, we propose TrackOcc, a cutting-edge approach that processes image inputs in a streaming, end-to-end manner with 4D panoptic queries to address the proposed task. Leveraging the localization-aware loss, TrackOcc enhances the accuracy of 4D panoptic occupancy tracking without bells and whistles. Experimental results demonstrate that our method achieves state-of-the-art performance on the Waymo dataset. The source code will be released at https://github.com/Tsinghua-MARS-Lab/TrackOcc.

Authors

• Zhuoguang Chen
• Kenan Li
• Xiuyu Yang
• Tao Jiang
• Yiming Li 0003
• Hang Zhao 0021

Keywords

• Image segmentation
• Three-dimensional displays
• Accuracy
• Source coding
• Semantics
• Streaming media
• Cameras
• Object tracking
• Autonomous vehicles
• Whistle
• Pedestrian
• Point Cloud
• Bounding Box
• Classification Score
• 3D Volume
• Instance Segmentation
• Semantic Labels
• Segmentation Quality
• 3D Tracking
• Voxel Grid
• 3D Domain
• Multi-view Images
• Track Quality
• Types Of Queries
• Transformation Module
• Bipartite Matching
• Training Frames
• 3D Bounding Box