A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking

Zhang, Huijun; Tang, Chun-hong; Sun, Xiaoming; Fu, Longsheng

doi:10.3390/agronomy13061469

Cited by 11 publications

(4 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wan et al [8] used faster R-CNN for the multiple detection of fruits by means of an existing Fruits 360 dataset. Also, Zhang et al [9] were able to detect apples for picking using CNN with an accuracy of 99.4%. Gao et al [10] was able to detect apples with an accuracy of 91.67%.…”

Section: Literature Analysismentioning

confidence: 99%

In-Depth Evaluation of Automated Fruit Harvesting in Unstructured Environment for Improved Robot Design

Zeeshan,

Aized,

Riaz

2024

Machines

View full text Add to dashboard Cite

Using modern machines like robots comes with its set of challenges when encountered with unstructured scenarios like occlusion, shadows, poor illumination, and other environmental factors. Hence, it is essential to consider these factors while designing harvesting robots. Fruit harvesting robots are modern automatic machines that have the ability to improve productivity and replace labor for repetitive and laborious harvesting tasks. Therefore, the aim of this paper is to design an improved orange-harvesting robot for a real-time unstructured environment of orchards, mainly focusing on improved efficiency in occlusion and varying illumination. The article distinguishes itself with not only an efficient structural design but also the use of an enhanced convolutional neural network, methodologically designed and fine-tuned on a dataset tailored for oranges integrated with position visual servoing control system. Enhanced motion planning uses an improved rapidly exploring random tree star algorithm that ensures the optimized path for every robot activity. Moreover, the proposed machine design is rigorously tested to validate the performance of the fruit harvesting robot. The unique aspect of this paper is the in-depth evaluation of robots to test five areas of performance that include not only the accurate detection of the fruit, time of fruit picking, and success rate of fruit picking, but also the damage rate of fruit picked as well as the consistency rate of the robot picking in varying illumination and occlusion. The results are then analyzed and compared with the performance of a previous design of fruit harvesting robot. The study ensures improved results in most aspects of the design for performance in an unstructured environment.

show abstract

Section: Literature Analysismentioning

confidence: 99%

In-Depth Evaluation of Automated Fruit Harvesting in Unstructured Environment for Improved Robot Design

Zeeshan,

Aized,

Riaz

2024

Machines

View full text Add to dashboard Cite

show abstract

“…By analyzing the disparities in these views, the method calculates the positional data of a specific point of interest. This approach finds extensive applications in various domains such as robot navigation [14][15][16][17], simultaneous localization and mapping (SLAM) [18,19], industrial automation [20,21], and intelligent agriculture [22,23].…”

Section: Introductionmentioning

confidence: 99%

Locating Anchor Drilling Holes Based on Binocular Vision in Coal Mine Roadways

Lei,

Zhang,

Dong

et al. 2023

Mathematics

View full text Add to dashboard Cite

The implementation of roof bolt support within a coal mine roadway has the capacity to bolster the stability of the encompassing rock strata and thereby mitigate the potential for accidents. To enhance the automation of support operations, this paper introduces a binocular vision positioning method for drilling holes, which relies on the adaptive adjustment of parameters. Through the establishment of a predictive model, the correlation between the radius of the target circular hole in the image and the shooting distance is ascertained. Based on the structural model of the anchor drilling robot and the related sensing data, the shooting distance range is defined. Exploiting the geometric constraints inherent to adjacent anchor holes, the precise identification of anchor holes is detected by a Hough transformer with an adaptive parameter-adjusted method. On this basis, the matching of the anchor hole contour is realized by using linear slope and geometric constraints, and the spatial coordinates of the anchor hole center in the camera coordinate system are determined based on the binocular vision positioning principle. The outcomes of the experiments reveal that the method attains a positioning accuracy of 95.2%, with an absolute error of around 1.52 mm. When compared with manual operation, this technique distinctly enhances drilling accuracy and augments support efficiency.

show abstract

“…Active ranging relies on a specific light source to illuminate the target fruit and measures depth information by receiving reflected light with a photoelectric sensor, so it is susceptible to outdoor sunlight interference [24][25][26]. Passive ranging captures data under natural light conditions without the need for specific light sources and utilizes a localization algorithm to calculate depth information [27][28][29]. Binocular camera stereo vision localization is a mainstream passive ranging method that is widely used in the visual systems of picking robots due to its advantages of low cost, high localization accuracy, and no additional power consumption compared to other localization methods.…”

Section: Introductionmentioning

confidence: 99%

Tomato Recognition and Localization Method Based on Improved YOLOv5n-seg Model and Binocular Stereo Vision

Zheng,

Liu,

Weng

et al. 2023

Agronomy

View full text Add to dashboard Cite

Recognition and localization of fruits are key components to achieve automated fruit picking. However, current neural-network-based fruit recognition algorithms have disadvantages such as high complexity. Traditional stereo matching algorithms also have low accuracy. To solve these problems, this study targeting greenhouse tomatoes proposed an algorithm framework based on YOLO-TomatoSeg, a lightweight tomato instance segmentation model improved from YOLOv5n-seg, and an accurate tomato localization approach using RAFT-Stereo disparity estimation and least squares point cloud fitting. First, binocular tomato images were captured using a binocular camera system. The left image was processed by YOLO-TomatoSeg to segment tomato instances and generate masks. Concurrently, RAFT-Stereo estimated image disparity for computing the original depth point cloud. Then, the point cloud was clipped by tomato masks to isolate tomato point clouds, which were further preprocessed. Finally, a least squares sphere fitting method estimated the 3D centroid co-ordinates and radii of tomatoes by fitting the tomato point clouds to spherical models. The experimental results showed that, in the tomato instance segmentation stage, the YOLO-TomatoSeg model replaced the Backbone network of YOLOv5n-seg with the building blocks of ShuffleNetV2 and incorporated an SE attention module, which reduced model complexity while improving model segmentation accuracy. Ultimately, the YOLO-TomatoSeg model achieved an AP of 99.01% with a size of only 2.52 MB, significantly outperforming mainstream instance segmentation models such as Mask R-CNN (98.30% AP) and YOLACT (96.49% AP). The model size was reduced by 68.3% compared to the original YOLOv5n-seg model. In the tomato localization stage, at the range of 280 mm to 480 mm, the average error of the tomato centroid localization was affected by occlusion and sunlight conditions. The maximum average localization error was ±5.0 mm, meeting the localization accuracy requirements of the tomato-picking robots. This study developed a lightweight tomato instance segmentation model and achieved accurate localization of tomato, which can facilitate research, development, and application of fruit-picking robots.

show abstract

A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking

Cited by 11 publications

References 40 publications

In-Depth Evaluation of Automated Fruit Harvesting in Unstructured Environment for Improved Robot Design

In-Depth Evaluation of Automated Fruit Harvesting in Unstructured Environment for Improved Robot Design

Locating Anchor Drilling Holes Based on Binocular Vision in Coal Mine Roadways

Tomato Recognition and Localization Method Based on Improved YOLOv5n-seg Model and Binocular Stereo Vision

Contact Info

Product

Resources

About