Fruit detection, segmentation and 3D visualisation of environments in apple orchards

Kang, Hanwen; Chen, Chao

doi:10.1016/j.compag.2020.105302

Cited by 164 publications

(79 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…An improved light-weight one-stage instance segmentation network ’Mobile-DasNet’ is developed in this research work, to perform fruit recognition, as shown in Figure 3 . Compared to the previous network, DasNet [ 28 ], which applies resnet-50 [ 47 ] as the backbone and a three levels Feature Pyramid Network (FPN), the proposed Mobile-DasNet applies a light-weigth backbone ’MobileNet’ [ 48 ] and a two-levels FPN (receive feature maps from C4, and C5 levels) to improve its computational efficiency. The proposed Mobile-DasNet achieves a weight size of 20.5 MB and the average running speed of 63 FPS on an NVIDIA GTX-1070 GPU.…”

Section: Methodsmentioning

confidence: 99%

“…Kang et al [ 27 ] introduced a novel multi-function neural network DasNet-v1 based on YOLO for real-time detection and semantic segmentation for both apples and branches in orchard environments. The detection and segmentation network with ResNet-101 backbone outperformed the corresponding task, while the network model with lightweight backbone also showed the best computation efficiency in the results.In the ensuing work [ 28 ], an enhanced deep neural network DasNet-v2 was developed, which achieved detection and instance segmentation on fruit and semantic segmentation on branches. The DasNet-v2 outperformed the previous neural network on the precision of apple detection and accuracy of semantic segmentation of branches and also applied instance segmentation on fruit as a new feature.…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

Kang

Zhou

Wang

et al. 2020

Sensors

Self Cite

135

View full text Add to dashboard Cite

Robotic harvesting shows a promising aspect in future development of agricultural industry. However, there are many challenges which are still presented in the development of a fully functional robotic harvesting system. Vision is one of the most important keys among these challenges. Traditional vision methods always suffer from defects in accuracy, robustness, and efficiency in real implementation environments. In this work, a fully deep learning-based vision method for autonomous apple harvesting is developed and evaluated. The developed method includes a light-weight one-stage detection and segmentation network for fruit recognition and a PointNet to process the point clouds and estimate a proper approach pose for each fruit before grasping. Fruit recognition network takes raw inputs from RGB-D camera and performs fruit detection and instance segmentation on RGB images. The PointNet grasping network combines depth information and results from the fruit recognition as input and outputs the approach pose of each fruit for robotic arm execution. The developed vision method is evaluated on RGB-D image data which are collected from both laboratory and orchard environments. Robotic harvesting experiments in both indoor and outdoor conditions are also included to validate the performance of the developed harvesting system. Experimental results show that the developed vision method can perform highly efficient and accurate to guide robotic harvesting. Overall, the developed robotic harvesting system achieves 0.8 on harvesting success rate and cycle time is 6.5 s.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

Kang

Zhou

Wang

et al. 2020

Sensors

Self Cite

135

View full text Add to dashboard Cite

show abstract

“…The works of [17] and [18] applied FCN model to detect the guava fruits and cotton, respectively. In our previous works [19,20,21], a YOLO and SPRNet [22] architecture based multi-purpose deep CNN network model DASNet was developed to perform real-time detection, instance segmentation of fruits, and semantic segmentation of branch/trunk in orchard environments. More similar works of deep-learning based algorithms in machine vision in agriculture applications can also be found in the recent survey [23].…”

Section: Related Work A: Visual Perceptionmentioning

confidence: 99%

“…The workflow of the visual perception and modelling algorithm is shown in Figure 1. Firstly, DASNet [20,21] performs segmentation and detection on input RGB images to extract objects of interest from the working scenarios. Then, by combining the depth map which is collected by RGB-D camera, the processed information is used to modelling the working scenarios of the orchards.…”

Section: System Architecturementioning

confidence: 99%

“…Each output branch predicts the class, boundary box, and object mask for each of the objects. The overall design of the detection branch of DASNet is shown in Figure 2 and more details of DASNet model can refer to the reference [21]. DASNet utilises Resnet-50 as backbone since previous of our works suggest that Resnet-50 achieves the balance between performance and computation efficiency.…”

Section: System Architecturementioning

confidence: 99%

See 1 more Smart Citation

Visual Perception and Modeling for Autonomous Apple Harvesting

2020

Self Cite

View full text Add to dashboard Cite

Vision perception and modelling are the essential tasks of robotic harvesting in the unstructured orchard. This paper develops a framework of visual perception and modelling for robotic harvesting of fruits in the orchard environments. The developed framework includes visual perception, scenarios mapping, and fruit modelling. The Visual perception module utilises a deep-learning model to perform multipurpose visual perception task within the working scenarios; The scenarios mapping module applies OctoMap to represent the multiple classes of objects or elements within the environment; The fruit modelling module estimates the geometry property of objects and estimates the proper access pose of each fruit. The developed framework is implemented and evaluated in the apple orchards. The experiment results show that visual perception and modelling algorithm can accurately detect and localise the fruits, and modelling working scenarios in real orchard environments. The F 1 score and mean intersection of union of visual perception module on fruit detection and segmentation are 0.833 and 0.852, respectively. The accuracy of the fruit modelling in terms of centre localisation and pose estimation are 0.955 and 0.923, respectively. Overall, an accurate visual perception and modelling algorithm are presented in this paper.

show abstract

Boosting Cost‐Efficiency in Robotics: A Distributed Computing Approach for Harvesting Robots

Xie,

Li,

Feng

et al. 2024

Journal of Field Robotics

View full text Add to dashboard Cite

Multi‐arm harvesting robots offer a promising solution to the labor shortage in fruit harvesting, due to their ability to improve harvesting efficiency. However, multi‐arm harvesters necessitate additional visual sensors to acquire distribution information of fruits within larger working spaces. Greater demands are consequently imposed on graphics computation, leading to increased costs in computing hardware of robot system. To balance the graphics computing cost and reduce energy consumption, distributed graphics computation frameworks for multi‐arm robot vision system are proposed in this study. First, a host‐edge framework is proposed to assign the tasks of image inference and depth alignment to host computer and edge computing modules through a decentralized mode of local connection. Moreover, to increase the endurance time of robot in application, the edge computing modules are reduced and the fifth generation mobile communication is integrated into robot graphics computing system to transfer on‐board image processing to a remote computing server with MQTT protocol. To verify the effectiveness of the proposed framework, comprehensive experiments were performed, demonstrating that, compared with traditional computing framework, the proposed local distributed framework reduced 35.6% average time consumption, and over 20 FPS average processing speed can be achieve. The remote distributed framework has reduced the computational power consumption of the on‐board system by approximately 23.1% while ensuring the performance is not lower than the local distributed framework. Finally, by discussing the two frameworks in terms of stability and cost, we present the commercial viability for the application of multi‐arm harvesting robot.

show abstract

Fruit detection, segmentation and 3D visualisation of environments in apple orchards

Cited by 164 publications

References 35 publications

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

Visual Perception and Modeling for Autonomous Apple Harvesting

Boosting Cost‐Efficiency in Robotics: A Distributed Computing Approach for Harvesting Robots

Contact Info

Product

Resources

About