Deep Learning-Based Masonry Wall Image Analysis

Abstract

In this paper we introduce a novel machine learning-based fully automatic approach for the semantic analysis and documentation of masonry wall images, performing in parallel automatic detection and virtual completion of occluded or damaged wall regions, and brick segmentation leading to an accurate model of the wall structure. For this purpose, we propose a four-stage algorithm which comprises three interacting deep neural networks and a watershed transform-based brick outline extraction step. At the beginning, a U-Net-based sub-network performs initial wall segmentation into brick, mortar and occluded regions, which is followed by a two-stage adversarial inpainting model. The first adversarial network predicts the schematic mortar-brick pattern of the occluded areas based on the observed wall structure, providing in itself valuable structural information for archeological and architectural applications. The second adversarial network predicts the pixels’ color values yielding a realistic visual experience for the observer. Finally, using the neural network outputs as markers in a watershed-based segmentation process, we generate the accurate contours of the individual bricks, both in the originally visible and in the artificially inpainted wall regions. Note that while the first three stages implement a sequential pipeline, they interact through dependencies of their loss functions admitting the consideration of hidden feature dependencies between the different network components. For training and testing the network a new dataset has been created, and an extensive qualitative and quantitative evaluation versus the state-of-the-art is given. The experiments confirmed that the proposed method outperforms the reference techniques both in terms of wall structure estimation and regarding the visual quality of the inpainting step, moreover it can be robustly used for various different masonry wall types.

On Using Perceptual Loss within the U-Net Architecture for the Semantic Inpainting of Textile Artefacts with Traditional Motifs

“…Particularly for CH conservation, a GAN method was employed for art painting restoration [12]. Also, looking more into the archaeological side, a pre-processing U-Net, followed by two GANs -one for segment completion, the other for color estimation -are used in [13] for the restoration of damaged walls.…”

Connecting Geometry and Semantics via Artificial Intelligence: From 3d Classification of Heritage Data to H-Bim Representations

“…In the architectural heritage domain, a very active research topic that has increasingly emerged in recent years involves the application of ML and DL techniques, fields of AI, to assist the digital data interpretation, logical organization, and semantic enrichment of a given asset being studied (Fiorucci et al, 2020) e.g. in terms of recognition of architectural elements , re-assembly of dismantled fragments (Paumard et al, 2020) and detection of occluded or damaged wall regions (Ibrahim et al, 2020).. In the case of survey data, the attention is devoted to the possibility to recognize and annotate, in much straightforward manner, the specific characteristics of a building or site: from the former experiments geared towards the semantic segmentation of heritage 2D images (Manfredi et al, 2013;Korc and Forstner, 2009;Ibrahim et al, 2020), the investigations moved on to studying more automatic annotations directly on 3D media, i.e.…”

“…in terms of recognition of architectural elements , re-assembly of dismantled fragments (Paumard et al, 2020) and detection of occluded or damaged wall regions (Ibrahim et al, 2020).. In the case of survey data, the attention is devoted to the possibility to recognize and annotate, in much straightforward manner, the specific characteristics of a building or site: from the former experiments geared towards the semantic segmentation of heritage 2D images (Manfredi et al, 2013;Korc and Forstner, 2009;Ibrahim et al, 2020), the investigations moved on to studying more automatic annotations directly on 3D media, i.e. point clouds (Grilli et al, 2018) and/or textured polygonal meshes .…”

Masonry Structure Analysis, Completion and Style Transfer Using a Deep Neural Network

“…The initial steps of the presented approach have been described in our prior papers [4][5][6], which methods we extend here with further steps and real world application examples. More specifically, [4] focused purely on the brick segmentation step, and [6] described an initial occlusion detection and inpainting model that was verified mostly on synthetic data samples.…”

“…More specifically, [4] focused purely on the brick segmentation step, and [6] described an initial occlusion detection and inpainting model that was verified mostly on synthetic data samples. The previously mentioned steps have been first used together and evaluated on real data in [5]. In the present paper, we introduce an extended model, additional experiments and novel use case options for wall image analysis with the proposed approach.…”