Recognition and Counting of Apples in a Dynamic State Using a 3D Camera and Deep Learning Algorithms for Robotic Harvesting Systems

Abstract

Recognition and 3D positional estimation of apples during harvesting from a robotic platform in a moving vehicle are still challenging. Fruit clusters, branches, foliage, low resolution, and different illuminations are unavoidable and cause errors in different environmental conditions. Therefore, this research aimed to develop a recognition system based on training datasets from an augmented, complex apple orchard. The recognition system was evaluated using deep learning algorithms established from a convolutional neural network (CNN). The dynamic accuracy of the modern artificial neural networks involving 3D coordinates for deploying robotic arms at different forward-moving speeds from an experimental vehicle was investigated to compare the recognition and tracking localization accuracy. In this study, a Realsense D455 RGB-D camera was selected to acquire 3D coordinates of each detected and counted apple attached to artificial trees placed in the field to propose a specially designed structure for ease of robotic harvesting. A 3D camera, YOLO (You Only Look Once), YOLOv4, YOLOv5, YOLOv7, and EfficienDet state-of-the-art models were utilized for object detection. The Deep SORT algorithm was employed for tracking and counting detected apples using perpendicular, 15°, and 30° orientations. The 3D coordinates were obtained for each tracked apple when the on-board camera in the vehicle passed the reference line and was set in the middle of the image frame. To optimize harvesting at three different speeds (0.052 ms−1, 0.069 ms−1, and 0.098 ms−1), the accuracy of 3D coordinates was compared for three forward-moving speeds and three camera angles (15°, 30°, and 90°). The mean average precision (mAP@0.5) values of YOLOv4, YOLOv5, YOLOv7, and EfficientDet were 0.84, 0.86, 0.905, and 0.775, respectively. The lowest root mean square error (RMSE) was 1.54 cm for the apples detected by EfficientDet at a 15° orientation and a speed of 0.098 ms−1. In terms of counting apples, YOLOv5 and YOLOv7 showed a higher number of detections in outdoor dynamic conditions, achieving a counting accuracy of 86.6%. We concluded that the EfficientDet deep learning algorithm at a 15° orientation in 3D coordinates can be employed for further robotic arm development while harvesting apples in a specially designed orchard.

A Seamless Deep Learning Approach for Apple Detection, Depth Estimation, and Tracking Using YOLO Models Enhanced by Multi-Head Attention Mechanism

“…The proposed model tracking experiment was performed on three different videos, and the consolidated MAPE applying DeepSORT was 0.197 and our proposed model attained a low MAPE of 0.027. The DeepSORT [56] technique also provided almost near results, but the duplication of apples and background apples that were not measured for the series of sequence counts made the model vulnerable. However, ByteTrack along with the recommended detection method categorized the apples in the foreground and background and included only the targeted apples in the count.…”

Intrarow Uncut Weed Detection Using You-Only-Look-Once Instance Segmentation for Orchard Plantations

“…A camera is also capable of distinguishing objects such as weeds and landmarks through image processing techniques [7]. The combination of deep learning algorithms with 3D cameras has made a significant contribution to object recognition, supporting robotic operations in orchards [6,8]. YOLO (You Only Look Once) is a highly efficient one-stage object detection model known for its speed, accuracy, and reliable real-time performance.…”

“…Even with a hand-held weed cutter, weeds around trees are still difficult to reach. In other cases, such as modern apple orchards, a V-shaped tree architecture is deployed to produce high-quality fruit and incorporates certain poles that can serve as obstacles for autonomous navigation [6]. A camera-attached small autonomous robotic weeder can be used to easily reach weeds while avoiding the obstacles present.…”

3D Camera and Single-Point Laser Sensor Integration for Apple Localization in Spindle-Type Orchard Systems

“…Different convolutional neural network (CNN)-based architectures, such as YOLOv3 [ 21 ], YOLOv5 [ 22 ], YOLOv7 [ 23 ], Faster RCNN [ 24 ], Mask RCNN [ 9 ], EfficientDet [ 25 ], and CenterNet, which have been trained based on apple datasets, have been used for detection and localization with high accuracy. Initially, 2D cameras were used as color sensors [ 26 , 27 ] to identify the apples, and the 2D information provided faced interference resulting from variations in light conditions.…”

“…Intel ® RealSense™ D435 [ 1 ] and D455™ (Intel Corporation, Santa Clara, CA, USA) [ 25 ] cameras were used to localize the apples, and the grasping pose was estimated based on the processing point cloud obtained from depth streams [ 42 ], but the results showed average accuracies of 0.61 cm and 4.80° degrees from the center position and orientation, respectively. The previous study [ 25 ] that we conducted was based on the state of art (SOTA) of detection algorithms: YOLOv4, YOLOv5, YOLOv7, and EfficientDet combined with a RealSense D455 camera to measure the accuracy of apple detection in terms of depth values at the dynamic stage. According to the results, we found that EfficientDet outperforms with higher accuracy than other networks as regards other detection models, compared with the RMSE values.…”