Semantic Segmentation of Natural Materials on a Point Cloud Using Spatial and Multispectral Features

Abstract

The characterization of natural spaces by the precise observation of their material properties is highly demanded in remote sensing and computer vision. The production of novel sensors enables the collection of heterogeneous data to get a comprehensive knowledge of the living and non-living entities in the ecosystem. The high resolution of consumer-grade RGB cameras is frequently used for the geometric reconstruction of many types of environments. Nevertheless, the understanding of natural spaces is still challenging. The automatic segmentation of homogeneous materials in nature is a complex task because there are many overlapping structures and an indirect illumination, so the object recognition is difficult. In this paper, we propose a method based on fusing spatial and multispectral characteristics for the unsupervised classification of natural materials in a point cloud. A high-resolution camera and a multispectral sensor are mounted on a custom camera rig in order to simultaneously capture RGB and multispectral images. Our method is tested in a controlled scenario, where different natural objects coexist. Initially, the input RGB images are processed to generate a point cloud by applying the structure-from-motion (SfM) algorithm. Then, the multispectral images are mapped on the three-dimensional model to characterize the geometry with the reflectance captured from four narrow bands (green, red, red-edge and near-infrared). The reflectance, the visible colour and the spatial component are combined to extract key differences among all existing materials. For this purpose, a hierarchical cluster analysis is applied to pool the point cloud and identify the feature pattern for every material. As a result, the tree trunk, the leaves, different species of low plants, the ground and rocks can be clearly recognized in the scene. These results demonstrate the feasibility to perform a semantic segmentation by considering multispectral and spatial features with an unknown number of clusters to be detected on the point cloud. Moreover, our solution is compared to other method based on supervised learning in order to test the improvement of the proposed approach.

Comparative Analysis of Algorithms to Cleanse Soil Micro-Relief Point Clouds

“…Although there are plenty of studies dealing with the performances of filtering or classification algorithms, most of them are predominantly geared towards larger scales [13,34,35] or focus on applications that require programming skills [12,30,36]. Bailey et al (2022) [37] demonstrate in their study that the performance of the different algorithms highly depends on the parameter settings.…”

Classification of 3D UAS-SfM Point Clouds in the Urban Environment

“…However, using new technologies to generate 3D UAS Structure-from-Motion (SfM) point clouds is time-efficient and cost-effective. Numerous studies have focused on classifying natural environments into primary classes (Vandapel et al, 2004;Lalonde et al, 2006;Brodu and Lague, 2012;Jurado et al, 2020). The present study focuses on classifying point clouds in heterogeneous urban environments.…”

Material Type Recognition of Indoor Scenes via Surface Reflectance Estimation

“…Based on extensive experimental evaluation, we suggest an optimal network structure for a multi-modal material type recognition. [43] propose a semantic segmentation method of natural materials on a point cloud using multi-spectral features. Xue et al [18] show improved performance on MINC-2500 (about 82.00% of classification) using deep encoding and pooling network called DEP-Net.…”