Visual Timing For Sound Source Depth Estimation in the Wild

Depth estimation enables a wide variety of 3D applications, such as robotics and autonomous driving. Despite significant work on various depth sensors, it is challenging to develop an all-in-one method to meet multiple basic criteria. In this paper, we propose a novel audio-visual learning scheme by integrating semantic features with physical spatial cues to boost monocular depth with only one microphone. Inspired by the flash-to-bang theory, we develop FBDepth, the first passive audio-visual depth estimation framework. It is based on the difference between the time-of-flight (ToF) of the light and the sound. We formulate sound source depth estimation as an audio-visual event localization task for collision events. To approach decimeter-level depth accuracy, we design a coarse-to-fine pipeline to push the temporary localization accuracy from event-level to millisecond-level by aligning audio-visual correspondence and manipulating optical flow. FBDepth feeds the estimated visual timestamp together with the audio clip and objects visual features to regress the source depth. We use a mobile phone to collect 3. 6K+ video clips with 24 different objects at up to 65m. FBDepth shows superior performance especially at a long range compared to monocular and stereo methods.

Authors

• Wei Sun
• Lili Qiu

Keywords

• Location awareness
• Visualization
• Accuracy
• Three-dimensional displays
• Depth measurement
• Robot sensing systems
• Sensors
• Synchronization
• Testing
• Sports
• Depth Estimation
• Sound Source
• Semantic Features
• Optical Flow
• Depth Camera
• Collision Events
• Audio Clips
• Background Noise
• Performance Metrics
• Intersection Over Union
• Raw Images
• Bounding Box
• Visual Input
• Target Object
• Depth Map
• Video Data
• Residual Block
• Slow Motion
• Sound Localization
• Audio Data
• Direction Of Arrival
• 1D Convolutional Layers
• Relative Absolute Error
• Changes In Acceleration
• Monocular Depth Estimation
• Stereo Camera
• Depth Perception
• Stereo Matching
• Motor Changes
• Motion Analysis