Semantic Segmentation for Building Façade 3d Point Cloud From 2d Orthophoto Images Using Transfer Learning

Abstract

Abstract. The task of semantic segmentation is an important one in the context of 3D building modelling. Indeed, developments in 3D generation techniques have rendered the point cloud ubiquitous. However pure data acquisition only captures geometric information and semantic classification remains to be performed, often manually, in order to give a tangible sense to the 3D data. Recently progress in computing power also opened the way for massive application of deep learning methods, including for semantic segmentation purposes. Although well established in the processing of 2D images, deep learning solutions remain an open question for 3D data. In this study, we aim to benefit from the vastly more developed 2D semantic segmentation by performing transfer learning on a photogrammetric orthoimage. The neural network was trained using labelled and rectified images of building façades. Another programme was then written to permit the passage between 2D orthoimage and 3D point cloud. Results show that the approach worked well and presents an alternative to help the automation process for point cloud semantic segmentation, at least in the case of photogrammetric data.

Towards Semantic Photogrammetry: Generating Semantically Rich Point Clouds from Architectural Close-Range Photogrammetry

“…The “shops” class refers to business signs and plaques. Note that this setup is more or less identical to a previous study as referred in [ 40 ].…”

“…In previous work detailed in Murtiyoso et al (2021) [ 40 ], we presented another approach to reprojection-based semantic segmentation. While the neural network was prepared in a similar manner, in this paper the pixel class prediction was performed on the orthophoto of a building facade instead of the input images as presented in this paper.…”

“…Three facades were then used to train the neural network, with the fourth and final facade used as a test data. In this case, the same main facade as the one used in this paper and in [ 40 ] was used.…”

“…Our previous work described in [ 40 ] presented an approach for the segmentation of 3D building facade based on orthophoto and the corresponding depth maps. The XY coordinates of each pixel in the orthophoto was used to determine the corresponding planimetric coordinates of the point in the point cloud and finally a winner-takes-all approach was applied to annotate the 3D points with the respective 2D pixel class.…”

“…Deep learning (DL), a subset of machine learning, has also seen major strides in performing semantic segmentation on 2D images [ 39 ]. Promising results can also be observed in 3D semantic segmentation, both indirectly [ 40 ] and directly [ 41 ]. It is worth noting that in DL-based solutions, a major bottleneck is the availability of labelled data for training.…”