Robust Fruit Counting: Combining Deep Learning, Tracking, and Structure from Motion

We present a novel fruit counting pipeline that combines deep segmentation, frame to frame tracking, and 3D localization to accurately count visible fruits across a sequence of images. Our pipeline works on image streams from a monocular camera, both in natural light, as well as with controlled illumination at night. We first train a Fully Convolutional Network (FCN) and segment video frame images into fruit and non-fruit pixels. We then track fruits across frames using the Hungarian Algorithm where the objective cost is determined from a Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) Tracker. In order to correct the estimated count from tracking process, we combine tracking results with a Structure from Motion (SfM) algorithm to calculate relative 3D locations and size estimates to reject outliers and double counted fruit tracks. We evaluate our algorithm by comparing with ground-truth human-annotated visual counts. Our results demonstrate that our pipeline is able to accurately and reliably count fruits across image sequences, and the correction step can significantly improve the counting accuracy and robustness. Although discussed in the context of fruit counting, our work can extend to detection, tracking, and counting of a variety of other stationary features of interest such as leaf-spots, wilt, and blossom.

Authors

• Xu Liu 0007
• Steven W. Chen
• Shreyas Aditya
• Nivedha Sivakumar
• Sandeep Dcunha
• Chao Qu
• Camillo J. Taylor
• Jnaneshwar Das
• Vijay Kumar 0001

Keywords

• Image segmentation
• Tracking
• Three-dimensional displays
• Pipelines
• Image sequences
• Deep learning
• Cameras
• Structure From Motion
• Fruit Count
• Convolutional Network
• Daylight
• Relative Size
• Image Frames
• 3D Position
• Correction Step
• Counting Accuracy
• Fully Convolutional Network
• Monocular Camera
• Visual Counting
• False Positive
• Support Vector Machine
• Cost Function
• 3D Reconstruction
• Image Regions
• Bounding Box
• Kalman Filter
• Optical Flow
• Unstructured Environments
• Scale-invariant Feature Transform
• Variable Distance
• Tree Rows
• Relative Depth
• Absolute Size
• Number Of Fruits
• Directions For Future Work
• Orange Trees