Weakly-Supervised Depth Completion during Robotic Micromanipulation from a Monocular Microscopic Image

Obtaining three-dimensional information, especially the z-axis depth information, is crucial for robotic micromanipulation. Due to the unavailability of depth sensors such as lidars in micromanipulation setups, traditional depth acquisition methods such as depth from focus or depth from defocus directly infer depth from microscopic images and suffer from poor resolution. Alternatively, micromanipulation tasks obtain accurate depth information by detecting the contact between an end-effector and an object (e. g. , a cell). Despite its high accuracy, only sparse depth data can be obtained due to its low efficiency. This paper aims to address the challenge of acquiring dense depth information during robotic cell micromanipulation. A weakly-supervised depth completion network is proposed to take cell images and sparse depth data obtained by contact detection as input to generate a dense depth map. A two-stage data augmentation method is proposed to augment the sparse depth data, and the depth map is optimized by a network refinement method. The experimental results show that the MAE value of the depth prediction error is less than 0. 3 µm, which proves the accuracy and effectiveness of the method. This deep learning network pipeline can be seamlessly integrated with the robotic micromanipulation tasks to provide accurate depth information.

Authors

• Han Yang
• Yufei Jin
• Guanqiao Shan
• Yibin Wang
• Yongbin Zheng
• Jiangfan Yu
• Yu Sun 0001
• Zhuoran Zhang 0001

Keywords

• Deep learning
• Accuracy
• Laser radar
• Microscopy
• Pipelines
• Robot sensing systems
• Data augmentation
• Microscopy Images
• Depth Completion
• Effective Method
• Cell Imaging
• Prediction Error
• Sparse Data
• Depth Map
• Depth Information
• Depth Camera
• Two-stage Method
• Depth Data
• Data Augmentation Methods
• Seamless Integration
• Refinement Network
• Dense Depth
• Cell Surface
• Root Mean Square Error
• Imaging Data
• Depth Values
• Holographic Microscopy
• 3D Space
• Visual Servoing
• Deformable Convolution
• Micropipette Tip
• Bilinear Interpolation
• Limited Amount Of Data
• Local Contact
• Mean Absolute Error
• Biological Cell Manipulation
• Automation at Micro/Nano Scales