Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

Abstract

Robotic harvesting shows a promising aspect in future development of agricultural industry. However, there are many challenges which are still presented in the development of a fully functional robotic harvesting system. Vision is one of the most important keys among these challenges. Traditional vision methods always suffer from defects in accuracy, robustness, and efficiency in real implementation environments. In this work, a fully deep learning-based vision method for autonomous apple harvesting is developed and evaluated. The developed method includes a light-weight one-stage detection and segmentation network for fruit recognition and a PointNet to process the point clouds and estimate a proper approach pose for each fruit before grasping. Fruit recognition network takes raw inputs from RGB-D camera and performs fruit detection and instance segmentation on RGB images. The PointNet grasping network combines depth information and results from the fruit recognition as input and outputs the approach pose of each fruit for robotic arm execution. The developed vision method is evaluated on RGB-D image data which are collected from both laboratory and orchard environments. Robotic harvesting experiments in both indoor and outdoor conditions are also included to validate the performance of the developed harvesting system. Experimental results show that the developed vision method can perform highly efficient and accurate to guide robotic harvesting. Overall, the developed robotic harvesting system achieves 0.8 on harvesting success rate and cycle time is 6.5 s.

Robotics Dexterous Grasping: The Methods Based on Point Cloud and Deep Learning

“…Llopart et al ( 2017 ), Zhou et al ( 2019 ), Yang et al ( 2020 ) and Zeng et al ( 2020 ) aim to learn grasping capability to accomplish opening doors, grabbing glasses, picking objects from human's hands and throwing arbitrary objects. More interestingly, Parhar et al ( 2018 ), Guo N. et al ( 2020 ), and Kang et al ( 2020 ) utilizes robot grasp ability to help completing crops harvesting in the farm. Zhang et al ( 2020 ) enables robot grasping to be controlled by a smartphone.…”

A Deep Learning-Based Vision System Combining Detection and Tracking for Fast On-Line Citrus Sorting

“…In recent years, deep learning has become state of the art due to its strong adaptability to variances within the working scene, showing potentials for a variety of tasks within machine vision such as image classification (He et al, 2016 ), object detection (Redmon and Farhadi, 2017 ), and image segmentation (Kang et al, 2020 ). As it is capable of automatically learning the image features, better recognition accuracy can be expected compared with traditional image processing methods (Kang and Chen, 2020a ).…”

Row End Detection and Headland Turning Control for an Autonomous Banana-Picking Robot

“…However, the process of picking fruits is still principally manually implemented, which leads to issues such as high labor intensity, high costs, and low efficiency [1,2]. To free the farmers from heavy work, reduce repetitive operations, and avoid the harm caused by certain operations, there is a great need for fruit-picking robots [3,4]. A fruit-picking robot is a type of robot that is designed to move through an entire field and pick fruits on its own [5][6][7][8][9][10][11][12][13].…”