Segmentation and 3D reconstruction of rose plants from stereoscopic images

Abstract

The method proposed in this paper is part of the vision module of a garden robot capable of navigating towards rose bushes and clip them according to a set of pruning rules. The method is responsible for performing the segmentation of the branches and recovering their morphology in 3D. The obtained reconstruction allows the manipulator of the robot to select the candidate branches to be pruned. This method first obtains a stereo pair of images and calculates the disparity image using block matching and the segmentation of the branches using a Fully Convolutional Neuronal Network modified to return a map with the probability at the pixel level of the presence of a branch. A post-processing step combines the segmentation and the disparity in order to improve the results. Then, the skeleton of the plant and the branching structure are calculated, and finally, the 3D reconstruction is obtained. The proposed approach is evaluated with five different datasets, three of them compiled by the authors and two from the state of the art, including indoor and outdoor scenes with uncontrolled environments. The different steps of the proposed pipeline are evaluated and compared with other state-of-the-art methods, showing that the accuracy of the segmentation improves other methods for this task, even with variable lighting, and also that the skeletonization and the reconstruction processes obtain robust results.

Branch Identification and Junction Points Location for Apple Trees Based on Deep Learning

“…Then, the skeleton structure of fruit trees can be extracted by skeleton extraction algorithm. Cuevas-Velasquez et al [22] adopted five algorithms, including Zhang & Suen, parallel thin, and medial axis, to extract branch skeleton. Among them, Zhang & Suen worked best, with F 1 of 91.06%.…”

“…Yang et al [21] used the branch segment merging algorithm to gain the citrus branches based on Mask R-CNN, with the identification of accuracy of 96.27%, and mapped the segmented RGB images to the depth images, with the error of the branch diameter of 1.17 mm. Cuevas-Velasquez et al [22] used Fully Convolutional Segmentation Network (FCSN) to segment the branches, with the branch segmentation accuracy of 88%, combined the result with the disparity image to conduct 3D reconstruction, and then finished the rose pruning through the help of the robotic arm. Based on U-Net, Liang et al [23] used the momentum optimization stochastic gradient descent method as the optimizer to segment the fruit and stem of litchi, effectively improving the segmentation accuracy to 95.54%.…”

Research on Outdoor Garden Scene Reconstruction Based on PMVS Algorithm

“…erefore, we can evaluate the method in this paper through this data set. Firstly, taking TB-Roses as the experimental object, four common segmentation methods such as DeepLabv3 [27], U-Net [28], FCSN [29], and SegNet [30] are evaluated, and their different superparameters are analyzed. Secondly, the postprocessing effect of parallax calculation and combination of segmented image and parallax image is evaluated.…”

Real-time Stereo Visual Servoing for Rose Pruning with Robotic Arm

“…The network outputs a binary image where it assigns a value of 1 to all the pixels that form part of a stem and 0 to the background. This network outperforms most of the state-of-the-art segmentations for the branch segmentation task [22]. The binary output is used to mask the disparity image to obtain only the disparities of the stems.…”