Point Cloud Semantic Segmentation Using a Deep Learning Framework for Cultural Heritage

Abstract

In the Digital Cultural Heritage (DCH) domain, the semantic segmentation of 3D Point Clouds with Deep Learning (DL) techniques can help to recognize historical architectural elements, at an adequate level of detail, and thus speed up the process of modeling of historical buildings for developing BIM models from survey data, referred to as HBIM (Historical Building Information Modeling). In this paper, we propose a DL framework for Point Cloud segmentation, which employs an improved DGCNN (Dynamic Graph Convolutional Neural Network) by adding meaningful features such as normal and colour. The approach has been applied to a newly collected DCH Dataset which is publicy available: ArCH (Architectural Cultural Heritage) Dataset. This dataset comprises 11 labeled points clouds, derived from the union of several single scans or from the integration of the latter with photogrammetric surveys. The involved scenes are both indoor and outdoor, with churches, chapels, cloisters, porticoes and loggias covered by a variety of vaults and beared by many different types of columns. They belong to different historical periods and different styles, in order to make the dataset the least possible uniform and homogeneous (in the repetition of the architectural elements) and the results as general as possible. The experiments yield high accuracy, demonstrating the effectiveness and suitability of the proposed approach.

A Hierarchical Machine Learning Approach for Multi-Level and Multi-Resolution 3D Point Cloud Classification

“…The dataset needs to be manually segmented by domain experts and must be broad enough to comprise enough classes for each case study, which in the CH context is quite tricky. Recently, a Dynamic Graph Convolutional Neural Network (DGCNN) supported by meaningful features (normal and HSV colours) has been employed in [39]. The DGCNN has been trained on the ArCH dataset [40], which includes 10 manually labelled point clouds subdivided into 11 classes.…”

A Benchmark for Large-Scale Heritage Point Cloud Semantic Segmentation

“…The authors also released a dataset with more than 10k images including categories like Altar, Apse, Belltower, Column, Dome (inner and outer), Flying buttress, Gargoyle, Stained glass, and Vault. In this context, several researchers have started to approach the topic of semantic segmentation of cultural heritage (CH) point clouds within the machine and deep learning framework (Grilli et al, 2019a;Kharroubi et al, 2019;Murtiyoso and Grussenmeyer, 2020;Pierdicca et al, 2020). However, the lack of an appropriate 3D heritage dataset does not allow an effective comparison between methods and results.…”

“…Moreover, as the classes included in these two standards are not enough to describe properly a CH, the AAT was perused and, within the Architectural elements class and Structural elements category, the 'Vaults', 'Arches' classes have been taken into account, whereas from Surface elements 'Moldings' have been selected. Following some studies and results of classification with the 3D features (Grilli et al, 2019b), it was decided to change the classification proposed in (Malinverni et al, 2019;Pierdicca et al, 2020), separating the class of columns and half-pilasters and inserting the latter in the new class 'Moldings' where there are also cornices and eaves. With this purpose, 9 classes have been selected (Figure 3), plus another one defined as 'Other', containing all the points not belonging to the previous classes (e.g.…”

“…If authors will submit the predicted results for a given point cloud (ideally for all), we will automatically compare the achieved results with the ground truth ones and provide results in terms of Overall Accuracy, F1 Score, Precision, Recall and Intersection over Union (IoU). Currently, the performances of state-of-art point cloud semantic segmentation networks are reported for PointNet (Qi et al, 2017a), PointNet++ (Qi et al, 2017b), PCNN (Atzmon et al, 2018), DGCNN (Wang et al, 2019) and modified DGCNN (Pierdicca et al, 2020). These DNNs are evaluated on 11 scenes out of 17 available.…”

Marker-Less Mobile Augmented Reality Application for Massive 3d Point Clouds and Semantics

“…Secondly, they permit to geometrically describe objects of any shape and scale * Corresponding author with an adapted resolution and point density. Finally, point clouds are generated by a wide variety of sensors permitting applications spanning from robotics (Parra et al, 2020) architecture, engineering and construction (Stojanovic et al, 2019) to cultural heritage (Pierdicca et al, 2020). For these unique properties, point clouds increasingly constitute a very important support for decision-making and support for semantic integration .…”