Ancient mural inpainting via structure information guided two-branch model

Abstract

Ancient murals are important cultural heritages for our exploration of ancient civilizations and are of great research value. Due to long-time exposure to the environment, ancient murals often suffer from damage (deterioration) such as cracks, scratches, corrosion, paint loss, and even large-region falling off. It is an urgent work to protect and restore these damaged ancient murals. Mural inpainting techniques refer to virtually filling the deteriorated regions by reconstructing the structure and texture elements of the mural images. Most existing mural inpainting approaches fail to fill loss contents that contain complex structures and diverse patterns since they neglect the importance of structure guidance. In this paper, we propose a structure-guided two-branch model based on the generative adversarial network (GAN) for ancient mural inpainting. In the proposed model, the mural inpainting process can be divided into two stages: structure reconstruction and content restoration. These two stages are conducted by using a structure reconstruction network (SRN) and a content restoration network (CRN), respectively. In the structure reconstruction stage, SRN employs the Gated Convolution and the Fast Fourier Convolution (FFC) residual block to reconstruct the missing structures of the damaged murals. In the content restoration stage, CRN uses the structures (generated by SRN) to guide the missing content restoration of the murals. We design a two-branch parallel encoder to improve the texture and color restoration quality for the missing regions of the murals. Moreover, we propose a cascade attention module that can capture long-term relevance information in the deep features. It helps to alleviate the texture-blur and color-bias problem. We conduct experiments on both simulated and real damaged murals, and compare our inpainting results with other four competitive approaches. Experimental results show that our proposed model outperforms other approaches in terms of texture clarity, color consistency and structural continuity of the restored mural images. In addition, the mural inpainting results of our model can achieve comparatively high quantitative evaluation metrics.

Extraction and restoration of scratched murals based on hyperspectral imaging—a case study of murals in the East Wall of the sixth grotto of Yungang Grottoes, Datong, China

“…Then the restoration result is improved based on improved exemplar-based region filling method. Experiments taken place in the mural of east wall of the sixth grotto of Yungang Grottoes and simulated mural show that the proposed method can achieve (20) SSIM x, y = (2u x u y + c 1 )(σ xy + c 2 ) (µ 2…”

“…Yurui et al [19] addressed extensive missing information on paintings and calligraphy by employing the concept of completing structure before filling in texture, conducting virtual restoration for large-scale missing damages. Deng et al [20] emphasized the significance of structural information in virtual restoration of mural by introducing a dual-branch image restoration model guided by structure. But this model still needs a large amount of real data of mural to train the net.…”

Chinese Ancient Paintings Inpainting Based on Edge Guidance and Multi-Scale Residual Blocks

“…Peng et al [34] proposed a model that uses dual-domain partial convolution to process valid pixels and combines frequency conversion to promote effective fusion of multi-scale features. Deng et al [35] believe that most existing mural inpainting models neglect the importance of structural guidance, making it impossible to fill in complex and diverse damaged content with structures. Thus, the authors proposed a structure-guided model based on GAN for the inpainting of ancient murals.…”

Chinese Ancient Paintings Inpainting Based on Edge Guidance and Multi-Scale Residual Blocks

“…In addition, the author also designed knowledge consistency attention to adaptively fuse attention scores and gradually refine the feature map. Deng et al [20] believed that most existing mural inpainting models neglected the importance of structural guidance, making it impossible to fill in complex and diverse damaged content with structures. Thus, the author proposed a structure-guided model based on GAN for the inpainting of ancient murals.…”