Segmentation of structural parts of rosebush plants with 3D point-based deep learning methods

Abstract

Background Segmentation of structural parts of 3D models of plants is an important step for plant phenotyping, especially for monitoring architectural and morphological traits. Current state-of-the art approaches rely on hand-crafted 3D local features for modeling geometric variations in plant structures. While recent advancements in deep learning on point clouds have the potential of extracting relevant local and global characteristics, the scarcity of labeled 3D plant data impedes the exploration of this potential. Results We adapted six recent point-based deep learning architectures (PointNet, PointNet++, DGCNN, PointCNN, ShellNet, RIConv) for segmentation of structural parts of rosebush models. We generated 3D synthetic rosebush models to provide adequate amount of labeled data for modification and pre-training of these architectures. To evaluate their performance on real rosebush plants, we used the ROSE-X data set of fully annotated point cloud models. We provided experiments with and without the incorporation of synthetic data to demonstrate the potential of point-based deep learning techniques even with limited labeled data of real plants. Conclusion The experimental results show that PointNet++ produces the highest segmentation accuracy among the six point-based deep learning methods. The advantage of PointNet++ is that it provides a flexibility in the scales of the hierarchical organization of the point cloud data. Pre-training with synthetic 3D models boosted the performance of all architectures, except for PointNet.

Improved Point-Cloud Segmentation for Plant Phenotyping Through Class-Dependent Sampling of Training Data to Battle Class Imbalance

“…An issue that was identified in previous work on point-cloud segmentation is that point clouds of plants typically have a high level of class imbalance, with, for instance, an abundance of leaf points, but only few points belonging to classes such as “flower” or “node” ( Boogaard et al, 2021 ; Turgut et al, 2022 ). This has consequences for semantic-segmentation methods, with the segmentation quality for the underrepresented classes lagging behind that of the overrepresented classes.…”

“…The highest Intersection-over-Union (IoU, a metric to measure segmentation quality) reported for these classes was 0.23, while for the majority class “leaf,” the IoU was 0.99. In point clouds of roses ( Turgut et al, 2022 ), the “stem” and “flower” were underrepresented as compared to the “leaf.” The highest observed IoU values for these underrepresented classes were 0.77 (“stem”) and 0.73 (flower), while the IoU for the overrepresented “leaf” was 0.95. In the current work, we propose a method to improve the segmentation of underrepresented classes, based on the level of class balance in the training data for the neural network.…”

“…They provided a large synthetic dataset of an urban environment in which the presence of small structures like poles and street furniture was increased, as compared to, for example, roads and pavements. In the plant domain, a synthetic dataset of sweet-pepper images including class and depth labels was generated by Barth et al (2018) and in the work of Turgut et al (2022) , synthetic 3D rosebush models were used to train neural networks to segment plants from the ROSE-X dataset ( Dutagaci et al, 2020 ). Such synthetic datasets could be modified to increase the focus on minority classes.…”

“…PointNet++ is one of the top performing neural network architectures for point cloud segmentation ( Guo et al, 2020 ). In the plant domain, PointNet++ was, for example, used in roses ( Turgut et al, 2022 ) and in cucumber ( Boogaard et al, 2021 ). In both works, it was found that PointNet++ was a suitable basis for semantic segmentation in the plant domain as compared to other deep neural networks.…”

Robustness of 3D point-based deep learning for plant organ segmentation against point density variation and noise

“…16 Moreover, the comparison with a traditional machine learning technique based on hand-crafted local surface features and Support Vector Machines (SVM) is performed. In our previous work, 18 six point-based deep learning networks were compared for plant segmentation, with PointNet++ yielding the highest performance. However, the 3D plant models used in the previous study were of high quality and relatively noise-free.…”