An automated fruit harvesting robot by using deep learning

Onishi, Yuki; Yoshida, Takeshi; Kurita, Hiroki; Fukao, Takanori; Arihara, Hiromu; Iwai, Ayako

doi:10.1186/s40648-019-0141-2

Cited by 156 publications

(71 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Recently, with the development of the deep learning architecture for 3D point cloud processing [ 7 , 34 ], more studies focus on grasping estimation by using the 3D data, such as Grasp Pose Detection (GPD) [ 35 ] and PointNet GPD [ 36 ]. In the agricultural cases, most of works [ 37 , 38 , 39 ] pick fruit by translating towards the targets, which cannot secure the success rate of harvesting in unstructured environments. Lehnert et al [ 40 ] applied a super-ellipsoid model to fit the sweep pepper and estimated the grasp pose by matching between the pre-defined shape and fruit.…”

Section: Literature Reviewmentioning

confidence: 99%

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

Kang

Zhou

Wang

et al. 2020

Sensors

135

View full text Add to dashboard Cite

Robotic harvesting shows a promising aspect in future development of agricultural industry. However, there are many challenges which are still presented in the development of a fully functional robotic harvesting system. Vision is one of the most important keys among these challenges. Traditional vision methods always suffer from defects in accuracy, robustness, and efficiency in real implementation environments. In this work, a fully deep learning-based vision method for autonomous apple harvesting is developed and evaluated. The developed method includes a light-weight one-stage detection and segmentation network for fruit recognition and a PointNet to process the point clouds and estimate a proper approach pose for each fruit before grasping. Fruit recognition network takes raw inputs from RGB-D camera and performs fruit detection and instance segmentation on RGB images. The PointNet grasping network combines depth information and results from the fruit recognition as input and outputs the approach pose of each fruit for robotic arm execution. The developed vision method is evaluated on RGB-D image data which are collected from both laboratory and orchard environments. Robotic harvesting experiments in both indoor and outdoor conditions are also included to validate the performance of the developed harvesting system. Experimental results show that the developed vision method can perform highly efficient and accurate to guide robotic harvesting. Overall, the developed robotic harvesting system achieves 0.8 on harvesting success rate and cycle time is 6.5 s.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

Kang

Zhou

Wang

et al. 2020

Sensors

135

View full text Add to dashboard Cite

show abstract

“…The purpose of the study was to develop an autonomous harvester system which can harvest any crop with a peduncle rather than damaging its flesh. Onishi et al [16] proposed a new system consisting of Single Shot MultiBox Detector (SSD) and a stereo camera for autonomous detection and harvesting of fruits. The experiment was conducted on an apple tree called "Fuji".…”

Section: Introductionmentioning

confidence: 99%

MNet: A Framework to Reduce Fruit Image Misclassification

Meshram¹,

Patil²,

Ramteke³

2021

ISI

View full text Add to dashboard Cite

Fast and accurate fruit classification is a major problem in the farming business. To achieve the same, the most popular technique used to build a classification model is "Transfer Learning", in which the weights of pretrained models are used in a new model to solve different but related problems. This technique assures the fast model building with a reduction in generalization error. After testing a popular image classification models namely, DenseNet161, InceptionV3, and MobileNetV2 on created dataset in which a "misclassification" is observed as a major problem which is overlooked by many researchers. This paper proposed a novel framework called "MNet: Merged Net" which not only improves the accuracy, but also addresses the misclassification problem. In this framework, the fruit classification problem is divided into small problems and build a separate model for each. In the final stage, the results of these models are combined. Two models called as FC_InceptionV3 (Fruit Classification InceptionV3) and MFC_InceptionV3 (Merged Fruit Classification InceptionV3) are created based on popular pretrained model InceptionV3. MFC_InceptionV3 is based on proposed framework. In this work, a dataset consisting of 12000 color images of top fruits in India with "Good" and "Bad" quality labels was created and published. The dataset consists of a total of 12 classes. The proposed framework MNet is tested on the most popular deep learning model called InceptionV3. The results of InceptionV3, FC_InceptionV3, and MFC_InceptionV3 are compared. The experimental results shows that the MFC_InceptionV3 model achieved 99.92% accuracy and moderates the image misclassification problem.

show abstract

“…Considering the challenges of the manipulator, the majority of these projects report issues regarding the speed and cost of the used robot. Instead of using a six degrees of freedom (6DOF) robotic arms like [2][3][4], or cartesian robots like [5,6], it is important for further research that alternative manipulators for robotic harvesting should be investigated to guarantee the profitability of the harvesting platform. However, this paper focusses on the gripping tool, and its challenges, itself.…”

Section: Introductionmentioning

confidence: 99%

“…Petterson et al developed a Bernoulli gripper that can adapt to 3D objects [10]. In contrast to the suction cup principle, Onishi et al [3] used a three-fingered gripper with yaws that enclose the fruit on multiple sides. This is comparable to the three-fingered gripper of Davidson et al; however, the latter integrated an extra compliant mechanism to enclose the apple depending on its shape [11][12][13].…”

Section: Introductionmentioning

confidence: 99%

Robotic Cultivation of Pome Fruit: A Benchmark Study of Manipulation Tools—From Research to Industrial Standards

et al. 2021

View full text Add to dashboard Cite

In pome fruit cultivation, apples and pears need to be handled in various processes such as harvesting and sorting. Currently, most processes require a vast amount of manual labor. Combined with a structural shortage of seasonal workers, innovation in this field is crucial. Automated processes could provide a solution wherein the search for an appropriate manipulation tool is essential. Aside from several grippers, customized for harvesting by various researchers, the industry also provides a wide variety of standardized manipulation tools. This paper benchmarks a wide set of the most relevant gripping principles, primarily based on their ability to successfully handle fruit, without causing damage. In addition, energy consumption and general feasibility are evaluated as well. The performed study showed that the customized foam gripper scores the overall best for all test scenarios at the cost of being the least energy efficient. Furthermore, most other gripping tools excelled at certain specific tasks rather than being generally deployable. Impactive grippers are better suited for harvesting at low energy consumption, while astrictive grippers are more suited for sorting tasks constricted by the available space. The results also showed that commercially available soft grippers are not always capable of handling sensitive fruits such as pears without causing damage.

show abstract

An automated fruit harvesting robot by using deep learning

Cited by 156 publications

References 12 publications

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

MNet: A Framework to Reduce Fruit Image Misclassification

Robotic Cultivation of Pome Fruit: A Benchmark Study of Manipulation Tools—From Research to Industrial Standards

Contact Info

Product

Resources

About