Enhancement and Restoration of Scratched Murals Based on Hyperspectral Imaging—A Case Study of Murals in the Baoguang Hall of Qutan Temple, Qinghai, China

Abstract

Environmental changes and human activities have caused serious degradation of murals around the world. Scratches are one of the most common issues in these damaged murals. We propose a new method for virtually enhancing and removing scratches from murals; which can provide an auxiliary reference and support for actual restoration. First, principal component analysis (PCA) was performed on the hyperspectral data of a mural after reflectance correction, and high-pass filtering was performed on the selected first principal component image. Principal component fusion was used to replace the original first principal component with a high-pass filtered first principal component image, which was then inverse PCA transformed with the other original principal component images to obtain an enhanced hyperspectral image. The linear information in the mural was therefore enhanced, and the differences between the scratches and background improved. Second, the enhanced hyperspectral image of the mural was synthesized as a true colour image and converted to the HSV colour space. The light brightness component of the image was estimated using the multi-scale Gaussian function and corrected with a 2D gamma function, thus solving the problem of localised darkness in the murals. Finally, the enhanced mural images were applied as input to the triplet domain translation network pretrained model. The local branches in the translation network perform overall noise smoothing and colour recovery of the mural, while the partial nonlocal block is used to extract the information from the scratches. The mapping process was learned in the hidden space for virtual removal of the scratches. In addition, we added a Butterworth high-pass filter at the end of the network to generate the final restoration result of the mural with a clearer visual effect and richer high-frequency information. We verified and validated these methods for murals in the Baoguang Hall of Qutan Temple. The results show that the proposed method outperforms the restoration results of the total variation (TV) model, curvature-driven diffusion (CDD) model, and Criminisi algorithm. Moreover, the proposed combined method produces better recovery results and improves the visual richness, readability, and artistic expression of the murals compared with direct recovery using a triple domain translation network.

Neural Networks for Hyperspectral Imaging of Historical Paintings: A Practical Review

“…There are several learning strategies that have so far been applied in the field of CH: supervised learning, unsupervised learning, and transfer learning [65,79,80]. In the first case, the available dataset consists of data points and their labels, and the NN algorithm learns a function which maps the input to the output by learning from available labelled examples, as observed in most cases [27,[43][44][45][46][47][48][49][53][54][55][56][57][58]60,62,63]. The second type of learning, unsupervised learning, is used to analyse and cluster unlabelled data [51,52,64].…”

“…Furthermore, in assisting the study of historical painting styles, Zhang et al have developed a strategy to extract the sketches of damaged or degraded paintings that also exploits spatial features [61]. In a recent advancement, Sun et al adopted a pre-trained model originally designed for photo restoration to virtually repair the scratched mural paintings that also extend the applicability of NN to digital restoration [63].…”

“…Some of the studies are based on XRF data, acquired in various experimental conditions (power source, acquisition time, and beam size) [50,51,59]. The RIS data cover several wavelength domains, from the most conventional VIS domain (380-750 nm [62], 400-700 nm [64], 400-720 nm [49]), extended to NIR (383-893 nm [56,57], 377-1033 nm [63], 377-1037 nm [61], 400-950 nm [53], 400-1000 nm [52,55], 822-1719 nm [54]), to the recently emerging SWIR range (930-2500 nm [52], 1000-2500 nm [27,58]).…”

“…However, historical data are more frequently used for the test of the developed network with the ultimate goal to study the material composition and distribution of historical artworks as illustrated for: one illuminated folio from the 15th century Book of Hours in [56]; two impressionistic paintings, the Poèmes Barbares by Paul Gauguin (1896) and The Bathers by Paul Cezanne (1899-1904) in [59]; a set of 11 paintings by the late Portuguese artist Amadeo de Souza-Cardoso in [49]; and an early Renaissance painting, Saint Michael Triumphs over the Devil by Bartolomé Bermejo [50]. In [63], a pre-trained model was tested on damaged murals of a Buddhist temple (c. 1392) to repair the scratches.…”

Extraction and Virtual Restoration of Scratched and Cracked Murals with Hyperspectral Data

“…divided the image into low frequency and high frequency through total variation (TV) model decomposition to enhance the structure and texture information on it. Based on hyperspectral images, Sun et al [3] used principal component analysis (PCA) and two-dimensional gamma transform to enhance the scratch information in the mural. Information extraction refers to the automatic extraction of deterioration on the image.…”