Virtual Cleaning of Works of Art Using Deep Convolutional Neural Networks

Abstract

Virtual cleaning of art is a key process that conservators apply to see the likely appearance of the work of art they have aimed to clean, before the process of cleaning. It is also of public interest, allowing people to see their favorite work of art without the yellow tint of the varnish which has covered the work. There have been many different approaches to virtually clean artwork, all of which need to physically clean the work in at least a few spots, an impediment in some cases. Another issue regarding the methods devised until now is that some of them need samples of pure white or black paint on the painting which might be, again, an obstacle as there are certainly works without any pure black and white paint. To overcome these shortcomings, a Convolutional Neural Network is trained on representative RGB images; the images are first artificially yellowed using a physics based model and the network is trained to go from the yellowed images to the original colored images. The results show improvement over a proposed method using a physical model to remove the impacts of age. The CNN is then applied to images of the Mona Lisa and The Virgin and Child with Saint Anne, both painted by Leonardo da Vinci, and works for which we have images from both before and after physical cleaning. Results show both a qualitative and quantitative improvement in the color quality of the resulting image. The novelty of the work proposed herein lies in two premises. First is the accuracy of the method, which is demonstrated through comparing the method with the only physical approach derived until now. Second is the generalizability of the method which is shown through blindly applying the method to two famous works of art for which no information but an RGB image of the uncleaned artwork is known.

Comparison of CoModGANs, LaMa and GLIDE for Art Inpainting- Completing M.C Escher's Print Gallery

“…For CoModGAN we used the demo provided 1 with the Places 2 dataset. For LaMa, we used the demo provided with the high-quality setting 2 . For GLIDE we used the Colab demo with a Guidance Scale of 4 and the text prompt "Print Gallery" 3 .…”

“…The traditional solution is to fine-tune these models and re-train them with images similar to the restored piece; this is typically also a challenge, as it can be difficult to provide large sets of examples of relevant artwork. Some examples of inpainting models specifically developed for art reconstruction include the works of Guptal et al [10] and Amiri and Messinger [2], which both propose models derived from computer vision inpainting and ex-tended to the art domain. Note that in both works domain experts were used to evaluate model performance, as an acknowledgment of the specific difficulty of evaluating inpainting in the specific context of art.…”