Deep learning-based apple detection using a suppression mask R-CNN

Chu, Pengyu; Li, Zhaojian; Lammers, Kyle; Lu, Renfu; Liu, Xiaoming

doi:10.1016/j.patrec.2021.04.022

Cited by 105 publications

(34 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using Mask-R-CNN to segment fruit images can distinguish fruits from occluded ones well. Chu et al (2021) used an integrated data set with two varieties of apple to train Mask-R-CNN for suppression. Jia et al (2020) optimized the Mask-R-CNN model in the backbone net, ROI layer, and FCN layer for apple harvesting robots.…”

Section: Convolutional Neural Network-based Fresh Fruit Detectionmentioning

confidence: 99%

Application of Convolutional Neural Network-Based Detection Methods in Fresh Fruit Production: A Comprehensive Review

et al. 2022

View full text Add to dashboard Cite

As one of the representative algorithms of deep learning, a convolutional neural network (CNN) with the advantage of local perception and parameter sharing has been rapidly developed. CNN-based detection technology has been widely used in computer vision, natural language processing, and other fields. Fresh fruit production is an important socioeconomic activity, where CNN-based deep learning detection technology has been successfully applied to its important links. To the best of our knowledge, this review is the first on the whole production process of fresh fruit. We first introduced the network architecture and implementation principle of CNN and described the training process of a CNN-based deep learning model in detail. A large number of articles were investigated, which have made breakthroughs in response to challenges using CNN-based deep learning detection technology in important links of fresh fruit production including fruit flower detection, fruit detection, fruit harvesting, and fruit grading. Object detection based on CNN deep learning was elaborated from data acquisition to model training, and different detection methods based on CNN deep learning were compared in each link of the fresh fruit production. The investigation results of this review show that improved CNN deep learning models can give full play to detection potential by combining with the characteristics of each link of fruit production. The investigation results also imply that CNN-based detection may penetrate the challenges created by environmental issues, new area exploration, and multiple task execution of fresh fruit production in the future.

show abstract

Section: Convolutional Neural Network-based Fresh Fruit Detectionmentioning

confidence: 99%

Application of Convolutional Neural Network-Based Detection Methods in Fresh Fruit Production: A Comprehensive Review

et al. 2022

View full text Add to dashboard Cite

show abstract

“…One of the key tasks in robotic apple harvesting is fruit detection and localization, where the former is to segment apples from the background areas whereas the latter subsequently calculates the 3D positions of the detected apples. In our preliminary work, a network with Mask R-CNN backbone and a suppression end was developed in [14] using a single RGB-D camera. In this new version, we extend the perception system to systematically fuse two RGB-D cameras to enhance the detection performance.…”

Section: B Multi-view Fusion For Robust Detection and Localizationmentioning

confidence: 99%

“…Once the images from the deep cameras are processed by the deep learning network, the bounding boxes of apple candidates are obtained. This suppression Mask R-CNN design has been reported in [14]. In the new perception version, we further fuse the detection results from the two camera channels using a fuzzy logic unit.…”

Section: B Multi-view Fusion For Robust Detection and Localizationmentioning

confidence: 99%

“…Second, we develop a deep learning-based multi-view fruit detection and localization framework by fusing two RGB-D sensors facing different angles. Compared to the single camera-based algorithm we developed earlier [14], this multi-view fusion offers enhanced performance in both detection and localization. Third, a unified planning algorithm that simultaneously optimizes picking sequence and dropping spots is developed, which significantly improves harvesting efficiency.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Algorithm Design and Integration for a Robotic Apple Harvesting System

Zhang¹,

Lammers²,

Chu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Due to labor shortage and rising labor cost for the apple industry, there is an urgent need for the development of robotic systems to efficiently and autonomously harvest apples. In this paper, we present a system overview and algorithm design of our recently developed robotic apple harvester prototype. Our robotic system is enabled by the close integration of several core modules, including calibration, visual perception, planning, and control. This paper covers the main methods and advancements in robust extrinsic parameter calibration, deep learning-based multi-view fruit detection and localization, unified picking and dropping planning, and dexterous manipulation control. Indoor and field experiments were conducted to evaluate the performance of the developed system, which achieved an average picking rate of 3.6 seconds per apple. This is a significant improvement over other reported apple harvesting robots with a picking rate in the range of 7-10 seconds per apple. The current prototype shows promising performance towards further development of efficient and automated apple harvesting technology. Finally, limitations of the current system and future work are discussed.This work is partly supported by the USDA-ARS inhouse project 5050-43640-003-00D.

show abstract

“…The detection algorithm based on Mask R-CNN achieved a fruit identification accuracy of 92.7% in the test dataset. A detailed report on the implementation of the mask R-CNN algorithm for apple detection and localization is given in [21]. Fig.…”

Section: A Visual Perceptionmentioning

confidence: 99%

System Design and Control of an Apple Harvesting Robot

Zhang

Lammers

Chu

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

There is a growing need for robotic apple harvesting due to decreasing availability and rising cost in labor. Towards the goal of developing a viable robotic system for apple harvesting, this paper presents synergistic mechatronic design and motion control of a robotic apple harvesting prototype, which lays a critical foundation for future advancements. Specifically, we develop a deep learning-based fruit detection and localization system using an RGB-D camera. A three degree-of-freedom manipulator is then designed with a hybrid pneumatic/motor actuation mechanism to achieve fast and dexterous movements. A vacuum-based end-effector is used for apple detaching. These three components are integrated into a robotic apple harvesting prototype with simplicity, compactness, and robustness. Moreover, a nonlinear velocity-based control scheme is developed for the manipulator to achieve accurate and agile motion control. Test experiments are conducted to demonstrate the performance of the developed apple harvesting robot.

show abstract

Deep learning-based apple detection using a suppression mask R-CNN

Cited by 105 publications

References 25 publications

Application of Convolutional Neural Network-Based Detection Methods in Fresh Fruit Production: A Comprehensive Review

Application of Convolutional Neural Network-Based Detection Methods in Fresh Fruit Production: A Comprehensive Review

Algorithm Design and Integration for a Robotic Apple Harvesting System

System Design and Control of an Apple Harvesting Robot

Contact Info

Product

Resources

About