Semantic segmentation and photogrammetry of crowdsourced images to monitor historic facades

Abstract

Crowdsourced images hold information could potentially be used to remotely monitor heritage sites, and reduce human and capital resources devoted to on-site inspections. This article proposes a combination of semantic image segmentation and photogrammetry to monitor changes in built heritage sites. In particular, this article focuses on segmenting potentially damaging plants from the surrounding stone masonry and other image elements. The method compares different backend models and two model architectures: (i) a one-stage model that segments seven classes within the image, and (ii) a two-stage model that uses the results from the first stage to refine a binary segmentation for the plant class. The final selected model can achieve an overall IoU of 66.9% for seven classes (54.6% for one-stage plant, 56.2% for two-stage plant). Further, the segmentation output is combined with photogrammetry to build a 3D segmented model to measure the area of biological growth. Lastly, the main findings from this paper are: (i) With the help of transfer learning and proper choice of model architecture, image segmentation can be easily applied to analyze crowdsourcing data. (ii) Photogrammetry can be combined with image segmentation to alleviate image distortions for monitoring purpose. (iii) Beyond the measurement of plant area, this method has the potential to be easily transferred into other tasks, such as monitoring cracks and erosion, or as a masking tool in the photogrammetry workflow.

3D Modeling & Analysis Techniques for the Apollo Temple in Delphi

“…Semantic segmentation, as per architectural, engineering, construction, and facility management requirements [59], remains an open subject with no one-size-fits-all resolution yet [57]. It extends beyond the scope of this manuscript, but current literature reveals that most researchers now consolidate towards a common pipeline for converting existing CH monuments to FEM-compatible configurations [15,[39][40][41][42][43]60,61].…”

“…Three-dimensional modeling and stone-by-stone segmentation from point clouds can be performed in several ways, which have been extensively researched [1,44,53]. State-ofthe-art approaches nowadays range from semi-automating recurrent label transmission using object-oriented programming integrated with commercial BIM architectural packages [55] to deep learning semantization utilizing either well-established methods in the field of computer vision, convolutional neural networks [53,56,57], or high-efficiency voxelbased feature engineering [58].…”

“…Of course, these polygon reduction schemes will always infer a representation accuracy drop, which is, to some extent, anticipated in FEM modeling. On the other hand, most sophisticated state-of-the-art implementations entail machine learning and even deep learning for class recognition and primitive extraction [62] that reach the stone level [53,57,59]. Contemporary research focuses on a shape-agnostic point cloud-driven segmentation [63] that facilitates primitive extraction, thus allowing for more comprehensive analysis, such as [59,64] or DEM [12].…”

Research on Deep Learning Based Genetic Intelligent Identification Method for Historical Buildings: A Case Study of Chinese Baroque Architecture in Harbin, China

“…Samhouri M and other scholars proposed an automatic multi category damage detection technology based on convolutional neural network (CNN) models for image classi cation and feature extraction, which is used to detect damage to historical structures, such as erosion, material loss, color changes in stones, and damage problems [24]. Scholars such as Liu ZW have combined semantic segmentation (DeeplabV3+) with drone photogrammetry methods for remote monitoring of changes in the historical architectural style of heritage sites, improving the e ciency of regular inspection and maintenance of historical architectural heritage sites, and reducing corresponding human and material resources [25]. Scholars such as Croce V have made full use of machine learning or deep learning, as well as network-based collaborative annotation platforms, to assist heritage experts in mixed annotation of building objects, such as identifying building components, degradation patterns, renovations, material mapping, etc., and to share and access 2D/3D information through the network, providing a more automated annotation tool for public and private stakeholders responsible for restoration and protection activities [26].…”

Quality Control of Hidden Construction Work Using Photographs Attached to Inspection Reports

“…Материалы и методы. В основу разрабатываемого метода контроля качества скрытых строительных работ положено применение фотограмметрии и различных методов цветотекстурного анализа фотографических изображений [12][13][14][15][16][17][18][19][20], и в частности контурного, пиксельного, макро-и микротекстурного анализа.…”