Algorithm Design and Integration for a Robotic Apple Harvesting System

Zhang, Kaixiang; Lammers, Kyle; Chu, Pengyu; Dickinson, N. J.; Li, Zhaojian; Lu, Renfu

doi:10.1109/iros47612.2022.9981417

Cited by 16 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1). An RGB camera with a resolution of 1280 × 720 was used to take images of apples at a distance of 1 − 2 meters from the tree trunks, which is the typical range of harvesting robots (De-An et al, 2011;Zhang et al, 2021Zhang et al, , 2022. The images were collected across multiple days to cover both cloudy and sunny weather conditions.…”

Section: Data Collection and Processingmentioning

confidence: 99%

O2RNet: Occluder-Occludee Relational Network for Robust Apple Detection in Clustered Orchard Environments

Chu¹,

Li²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Automated apple harvesting has attracted significant research interest in recent years due to its potential to revolutionize the apple industry, addressing the issues of shortage and high costs in labor. One key technology to fully enable efficient automated harvesting is accurate and robust apple detection, which is challenging due to complex orchard environments that involve varying lighting conditions and foliage/branch occlusions. Furthermore, clustered apples are common in the orchard, which brings additional challenges as the clustered apples may be identified as one apple. This will cause issues in localization for subsequent robotic operations. In this paper, we present the development of a novel deep learning-based apple detection framework, Occluder-Occludee Relational Network (O2RNet), for robust detection of apples in such clustered environments. This network exploits the occuluder-occludee relationship modeling head by introducing a feature expansion structure to enable the combination of layered traditional detectors to split clustered apples and foliage occlusions. More specifically, we collect a comprehensive apple orchard image dataset under different lighting conditions (overcast, front lighting, and back lighting) with frequent apple occlusions. We then develop a novel occlusion-aware network for apple detection, in which a feature expansion structure is incorporated into the convolutional neural networks to extract additional features generated by the original network for occluded apples. Comprehensive evaluations are performed, which show that the developed O2RNet outperforms state-of-the-art models with a higher accuracy of 94% and a higher F1-score of 0.88 on apple detection.

show abstract

Section: Data Collection and Processingmentioning

confidence: 99%

O2RNet: Occluder-Occludee Relational Network for Robust Apple Detection in Clustered Orchard Environments

Chu¹,

Li²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…According to the aforementioned picking procedure, it can be seen that the fruit detection and localization is a key task in automated apple harvesting. Our previous system prototypes [10,12] utilized RGB-D cameras to facilitate fruit detection and localization. However, laboratory and field tests found that the commercial RGB-D cameras could not provide accurate depth information of the target fruits under leaf/branch occlusions and/or challenging lighting conditions.…”

Section: Overview Of the Robotic Apple Harvesting Systemmentioning

confidence: 99%

“…Therefore, according to the evaluation results, it can be concluded that the ALACS unit can meet the requirements for fruit localization and can be integrated with other hardware modules for automated apple harvesting. The RealSense D435i RGB-D camera was used in our previous apple harvesting robotic prototypes to localize the fruit [10,12]. According to the manufacturer's datasheet [49], this camera offers a measurement accuracy of less than 2% of the depth range.…”

Section: Localization Accuracymentioning

confidence: 99%

“…Research on robotic harvesting technology has been ongoing for several decades, and different robotic systems have been attempted for semi-automated or fully automated fruit harvesting [2][3][4][5][6][7][8][9][10][11][12][13][14][15]. A typical robotic harvesting system consists of a perception module, a manipulator, and an end-effector.…”

Section: Introductionmentioning

confidence: 99%

“…Once the fruits are recognized and a picking sequence is determined (see e.g., [12]), three-dimensional (3D) localization needs to be conducted to compute the spatial coordinates of a target fruit. Accurate fruit localization is crucial since erroneous localization will cause the manipulator to miss the target and subsequently degrade the harvesting performance of the robotic system.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Active Laser-Camera Scanning for High-Precision Fruit Localization in Robotic Harvesting: System Design and Calibration

Zhang,

Chu,

Lammers

et al. 2023

Horticulturae

Self Cite

View full text Add to dashboard Cite

Robust and effective fruit detection and localization is essential for robotic harvesting systems. While extensive research efforts have been devoted to improving fruit detection, less emphasis has been placed on the fruit localization aspect, which is a crucial yet challenging task due to limited depth accuracy from existing sensor measurements in the natural orchard environment with variable lighting conditions and foliage/branch occlusions. In this paper, we present the system design and calibration of an Active LAser-Camera Scanner (ALACS), a novel perception module for robust and high-precision fruit localization. The hardware of the ALACS mainly consists of a red line laser, an RGB camera, and a linear motion slide, which are seamlessly integrated into an active scanning scheme where a dynamic-targeting laser-triangulation principle is employed. A high-fidelity extrinsic model is developed to pair the laser illumination and the RGB camera, enabling precise depth computation when the target is captured by both sensors. A random sample consensus-based robust calibration scheme is then designed to calibrate the model parameters based on collected data. Comprehensive evaluations are conducted to validate the system model and calibration scheme. The results show that the proposed calibration method can detect and remove data outliers to achieve robust parameter computation, and the calibrated ALACS system is able to achieve high-precision localization with the maximum depth measurement error being less than 4 mm at distance ranging from 0.6 to 1.2 m.

show abstract

An automated apple harvesting robot—From system design to field evaluation

Zhang,

Lammers,

Chu

et al. 2023

Journal of Field Robotics

View full text Add to dashboard Cite

Decreased availability and rising cost in labor poses a serious threat to the long‐term profitability and sustainability of the apple industry in the United States and many other countries. Harvest automation is thus urgently needed. In this paper, we present the unified system design and field evaluation of a new apple harvesting robot. The robot is mainly composed of a specially designed perception component, a four‐degree‐of‐freedom manipulator, an improved vacuum‐based soft end‐effector, and a dropping/catching component to receive and transport picked fruits. Software algorithms are developed to enable synergistic coordination of the hardware components for efficient, automated harvesting of apples in challenging orchard environments. Specifically, by integrating modified triangulation and image processing and analysis algorithms, a novel perception strategy is developed to achieve robust apple detection and precise localization. Improved planning and control algorithms are developed to guide the robot to the target positions. The performance of the robotic system was evaluated through field tests in two apple orchards with different tree architectures and foliage conditions. In the orchard where trees were young and well‐pruned, the robot achieved 82.4% successful harvesting rate. In a second, older orchard with dense, clustered branches and foliage, the robot had 65.2% successful rate. The average cycle time to harvest a fruit was approximately 6 s, which included software algorithm processing and hardware execution. Moreover, through an in‐depth analysis of the obtained results, limitations and planned future works are discussed.

show abstract

Algorithm Design and Integration for a Robotic Apple Harvesting System

Cited by 16 publications

References 16 publications

O2RNet: Occluder-Occludee Relational Network for Robust Apple Detection in Clustered Orchard Environments

O2RNet: Occluder-Occludee Relational Network for Robust Apple Detection in Clustered Orchard Environments

Active Laser-Camera Scanning for High-Precision Fruit Localization in Robotic Harvesting: System Design and Calibration

An automated apple harvesting robot—From system design to field evaluation

Contact Info

Product

Resources

About