Deep Semantic Segmentation of 3D Plant Point Clouds

Abstract

Plant phenotyping is an essential step in the plant breeding cycle, necessary to ensure food safety for a growing world population. Standard procedures for evaluating three-dimensional plant morphology and extracting relevant phenotypic characteristics are slow, costly, and in need of automation. Previous work towards automatic semantic segmentation of plants relies on explicit prior knowledge about the species and sensor set-up, as well as manually tuned parameters. In this work, we propose to use a supervised machine learning algorithm to predict per-point semantic annotations directly from point cloud data of whole plants and minimise the necessary user input. We train a PointNet++ variant on a fully annotated procedurally generated data set of partial point clouds of tomato plants, and show that the network is capable of distinguishing between the semantic classes of leaves, stems, and soil based on structural data only. We present both quantitative and qualitative evaluation results, and establish a proof of concept, indicating that deep learning is a promising approach towards replacing the current complex, laborious, species-specific, state-of-the-art plant segmentation procedures.

How to make sense of 3D representations for plant phenotyping: a compendium of processing and analysis techniques

“…Heiwolt et al [301] applied the PointNet++ architecture, adjusted for point-wise segmentation applications, on tomato plants and showed that this network was able to successfully predict per-point semantic annotations for soil, leaves, and stems directly from point cloud data.…”

“…Deep Learning (DL) Banana [90] Maize (Corn) [160,299,310,313] Rice [311] Rosebush [264] Rosette plants [1] Sorghum [276] Thale cress (Arabidopsis) [17] Tobacco [276] Tomato [2,160,276,301] DL is a very commonly employed algorithm in the ML algorithms, and it is derived from the conventional neural network but considerably outperforms its predecessors. DL employs transformations and graph technologies simultaneously in order to build up multi-layer learning models.…”

High‐throughput phenotyping for breeding targets—Current status and future directions of strawberry trait automation

“…Segmentation of plants into organs is an essential step prior to the extraction of phenotypic traits. Deep learning methods such as PointNet++ variants have shown promise for this in the context of tomato plants, performing 3D semantic segmentation of synthetically generated plants without prior knowledge of species and sensor setup (Heiwolt et al., 2021). In comparison to cereal crops, such as wheat, both the complex growth habit of the strawberry plants, as well as their relatively small size, increase the phenotyping complexity (Zheng et al., 2021).…”

3D data-augmentation methods for semantic segmentation of tomato plant parts

“…More recently, Chebrolu et al (2021) achieved leaf and stem segmentation using a support vector machine (SVM) based on fast point feature histograms (FPFH). However, these methods rely on human work to provide discriminative features for segmentation, a process which is sensitive to natural variations of plants even within a same cultivar, resulting in poor generalization (Heiwolt et al, 2021).…”