A real-time branch detection and reconstruction mechanism for harvesting robot via convolutional neural network and image segmentation

Wan, Hao; Zhao, Fan; Yu, Xiaojun; Kang, Meilin; Wang, Pengbo; Zeng, Xilei

doi:10.1016/j.compag.2021.106609

Cited by 27 publications

(10 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This will require robust state estimation and perception capabilities inside an unstructured environment, which is subject to extensive research [18,19]. Branch detection, which will be required to achieve an autonomous system, is strongly investigated in agricultural robotics [20][21][22], but usually requires strong computational power and is therefore challenging to accomplish on a lightweight and compact robot.…”

Section: Discussionmentioning

confidence: 99%

Multimodal aerial-tethered robot for tree canopy exploration

Kirchgeorg

Mintchev

2022

2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Forest canopies are the biggest habitat for terrestrial life, yet our understanding of environmental processes and biodiversity inside the canopy continues to be limited due to labour and resource intensive data collection. Existing aerial and climbing robots also struggle to access these complex environments, while animals easily navigate them using multiple means of locomotion. Following this insight we present a robot with multimodal mobility obtained by combining aerial and tethered locomotion. After the robot is deployed at the top of the tree, it can descend with the tether and maneuver around leaves and branches with its thrusters. The tether increases robustness and safety and allows for resting as well as emergency retrieval of the system. The aerial locomotion grants the system the ability to move in a conical 3D space constrained by the tether. We modelled the static system and validated the impact of design parameters on it. A simple control architecture for teleoperation is discussed and its performance is analyzed. The proposed multimodal mobility is demonstrated in preliminary outdoor tests, which show how our robot can move within the canopy while continuously monitoring the environment.

show abstract

Section: Discussionmentioning

confidence: 99%

Multimodal aerial-tethered robot for tree canopy exploration

Kirchgeorg

Mintchev

2022

2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…The methods in both (Kalampokas et al, 2021) and (Chen et al, 2021) can only segment the stem of a single cluster of grape or banana. Wan et al (2022) proposed a realtime branch detection and reconstruction method applied to fruit harvesting. To segment the branches separately, the authors first detect branch region boxes using YOLOv4, then utilize image segmentation to locate the branch boundaries.…”

Section: Introductionmentioning

confidence: 99%

“…The methods in both ( Kalampokas et al., 2021 ) and ( Chen et al., 2021 ) can only segment the stem of a single cluster of grape or banana. Wan et al. (2022) proposed a real-time branch detection and reconstruction method applied to fruit harvesting.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Grasping and cutting points detection method for the harvesting of dome-type planted pumpkin using transformer network-based instance segmentation architecture

et al. 2023

View full text Add to dashboard Cite

An accurate and robust keypoint detection method is vital for autonomous harvesting systems. This paper proposed a dome-type planted pumpkin autonomous harvesting framework with keypoint (grasping and cutting points) detection method using instance segmentation architecture. To address the overlapping problem in agricultural environment and improve the segmenting precision, we proposed a pumpkin fruit and stem instance segmentation architecture by fusing transformer and point rendering. A transformer network is utilized as the architecture backbone to achieve a higher segmentation precision and point rendering is applied so that finer masks can be acquired especially at the boundary of overlapping areas. In addition, our keypoint detection algorithm can model the relationships among the fruit and stem instances as well as estimate grasping and cutting keypoints. To validate the effectiveness of our method, we created a pumpkin image dataset with manually annotated labels. Based on the dataset, we have carried out plenty of experiments on instance segmentation and keypoint detection. Pumpkin fruit and stem instance segmentation results show that the proposed method reaches the mask mAP of 70.8% and box mAP of 72.0%, which brings 4.9% and 2.5% gains over the state-of-the-art instance segmentation methods such as Cascade Mask R-CNN. Ablation study proves the effectiveness of each improved module in the instance segmentation architecture. Keypoint estimation results indicate that our method has a promising application prospect in fruit picking tasks.

show abstract

“…Wan et al developed an improved YOLOV4 to detect branch segments, applied a thresholding segmentation method to remove background, and used a polynomial fit to reconstruct the branches. The detection F1 score was 90%, and the running speed was 22.7 frames per second (FPS) ( Wan et al, 2022 ). Because manually annotating a large empirical dataset is time-consuming and costly, Barth et al trained DeepLabV2 with VGG16 as the backbone on a large synthetic dataset and then fine-tuned DeepLabV2 on a small empirical dataset.…”

Section: Introductionmentioning

confidence: 99%

Real-time guava tree-part segmentation using fully convolutional network with channel and spatial attention

Lin

Chen

et al. 2022

Front. Plant Sci.

View full text Add to dashboard Cite

It is imminent to develop intelligent harvesting robots to alleviate the burden of rising costs of manual picking. A key problem in robotic harvesting is how to recognize tree parts efficiently without losing accuracy, thus helping the robots plan collision-free paths. This study introduces a real-time tree-part segmentation network by improving fully convolutional network with channel and spatial attention. A lightweight backbone is first deployed to extract low-level and high-level features. These features may contain redundant information in their channel and spatial dimensions, so a channel and spatial attention module is proposed to enhance informative channels and spatial locations. On this basis, a feature aggregation module is investigated to fuse the low-level details and high-level semantics to improve segmentation accuracy. A tree-part dataset with 891 RGB images is collected, and each image is manually annotated in a per-pixel fashion. Experiment results show that when using MobileNetV3-Large as the backbone, the proposed network obtained an intersection-over-union (IoU) value of 63.33 and 66.25% for the branches and fruits, respectively, and required only 2.36 billion floating point operations per second (FLOPs); when using MobileNetV3-Small as the backbone, the network achieved an IoU value of 60.62 and 61.05% for the branches and fruits, respectively, at a speed of 1.18 billion FLOPs. Such results demonstrate that the proposed network can segment the tree-parts efficiently without loss of accuracy, and thus can be applied to the harvesting robots to plan collision-free paths.

show abstract

A real-time branch detection and reconstruction mechanism for harvesting robot via convolutional neural network and image segmentation

Cited by 27 publications

References 18 publications

Multimodal aerial-tethered robot for tree canopy exploration

Multimodal aerial-tethered robot for tree canopy exploration

Grasping and cutting points detection method for the harvesting of dome-type planted pumpkin using transformer network-based instance segmentation architecture

Real-time guava tree-part segmentation using fully convolutional network with channel and spatial attention

Contact Info

Product

Resources

About