Deep learning for the extraction of sketches from spectral images of historical paintings

Abstract

Sketch extraction is of great value for historians to copy and study historical painting styles. However, most of the existing sketch extraction methods can successfully perform extraction only if the sketches are well preserved, but for paintings with severe conservation issues, the extraction methods need to be improved. Therefore, we propose a sketch extraction method using spectral imaging and deep learning. Firstly, the spectral image data is collected and the bands sensitive to the sketches are extracted by using the prior knowledge of the sketches (e.g. near infrared bands will be chosen if the sketches are made of carbon ink). A publicly available image dataset of natural scenes is used to pre-train the bi-directional cascade network (BDCN). The network parameters in the model are then fine-tuned by using the sketches drawn by experts based on images of painted cultural objects, so as to solve the problem of insufficient sketch dataset of painted cultural objects and enhance the generalization ability of the model. Finally, the U-Net is used to further suppress unwanted information, to make the sketch clearer. Experimental results show that the proposed method can extract clear sketches even with faded paintings and the presence of unwanted information or instrumental noise. It is superior to the other six advanced extraction methods in visual and objective comparison. The proposed deep learning method is also compared with an unsupervised clustering method using Self-Organising Map (SOM) which is a 'shallow learning' method where pixels of similar spectra are grouped into clusters without the need for data labeling by experts.

Neural Networks for Hyperspectral Imaging of Historical Paintings: A Practical Review

“…Research conducted by Striova et al applied the NN to improve the visibility of the pentimenti and underdrawing style based on VNIR multispectral data acquired on a pair of homonymous paintings by Manet and Titian, respectively [60]. Furthermore, in assisting the study of historical painting styles, Zhang et al have developed a strategy to extract the sketches of damaged or degraded paintings that also exploits spatial features [61]. In a recent advancement, Sun et al adopted a pre-trained model originally designed for photo restoration to virtually repair the scratched mural paintings that also extend the applicability of NN to digital restoration [63].…”

“…Some of the studies are based on XRF data, acquired in various experimental conditions (power source, acquisition time, and beam size) [50,51,59]. The RIS data cover several wavelength domains, from the most conventional VIS domain (380-750 nm [62], 400-700 nm [64], 400-720 nm [49]), extended to NIR (383-893 nm [56,57], 377-1033 nm [63], 377-1037 nm [61], 400-950 nm [53], 400-1000 nm [52,55], 822-1719 nm [54]), to the recently emerging SWIR range (930-2500 nm [52], 1000-2500 nm [27,58]).…”

“…Two different types of complex CNN architectures have so far been employed to analyse hyperspectral data from painted artworks: the first one to perform sketch extraction [61], and the second one to identify pure pigments [49].…”

“…A carefully designed loss function making use of a distinction between edge and non-edge pixels was employed. To account for the imbalanced distribution of edge and non-edge pixels, a class-balanced cross-entropy loss was employed [61]. Due to insufficient training data, the authors resorted to transfer learning by pre-training the BDCN model on a publicly available dataset of natural scenes.…”

“…The second type of learning, unsupervised learning, is used to analyse and cluster unlabelled data [51,52,64]. In the case of transfer learning, the model is pre-trained with a large dataset and then fine-tuned with a separate smaller dataset [50,59,61]. This type of learning is discussed in detail in Section 5.3.2.…”