Art authentication with vision transformers

Abstract

In recent years, transformers, initially developed for language, have been successfully applied to visual tasks. Vision transformers have been shown to push the state of the art in a wide range of tasks, including image classification, object detection, and semantic segmentation. While ample research has shown promising results in art attribution and art authentication tasks using convolutional neural networks, this paper examines whether the superiority of vision transformers extends to art authentication, improving, thus, the reliability of computer-based authentication of artworks. Using a carefully compiled dataset of authentic paintings by Vincent van Gogh and two contrast datasets, we compare the art authentication performances of Swin transformers with those of EfficientNet. Using a standard contrast set containing imitations and proxies (works by painters with styles closely related to van Gogh), we find that EfficientNet achieves the best performance overall. With a contrast set that only consists of imitations, we find the Swin transformer to be superior to EfficientNet by achieving an authentication accuracy of over 85%. These results lead us to conclude that vision transformers represent a strong and promising contender in art authentication, particularly in enhancing the computer-based ability to detect artistic imitations.

Synthetic images aid the recognition of human-made art forgeries

“…First, we test whether the addition of each of the synthetic sets separately, as well as the combination of Stable Diffusion and fine-tuned GANs, improves the classification accuracy of the human-made forgeries against a baseline trained using ‘proxies’ and ‘imitations’. This baseline also agrees with the previous work [ 18 ].…”

“…All images are pre-processed according to the procedure detailed in [ 18 ]. Specifically, we generate sub-images of paintings, i.e., RGB images normalized to a fixed size of 256 × 256 pixels, with channel values normalized to the unit interval.…”

“…After preparing the training and testing datasets of human-made and synthetic artwork forgeries as described in the previous Section, we proceed to explain the classification methodology on the gathered dataset. In line with the approach outlined in [ 18 ], we employ transformer-based classification methods (specifically, Swin Base [ 31 ]) and state-of-the-art Convolutional Neural Networks (EfficientNet B0 [ 32 ]) to distinguish between authentic artworks and forgeries. These models have been adopted in [ 18 ] and proved to outperform the canonical ResNet101 model, the typical baseline model for art authentication.…”

“…In line with the approach outlined in [ 18 ], we employ transformer-based classification methods (specifically, Swin Base [ 31 ]) and state-of-the-art Convolutional Neural Networks (EfficientNet B0 [ 32 ]) to distinguish between authentic artworks and forgeries. These models have been adopted in [ 18 ] and proved to outperform the canonical ResNet101 model, the typical baseline model for art authentication. The Swin Base is an image transformer model that uses a hierarchical structure with shifting windows to reduce the computational complexity of transformer models, it accepts inputs of size 224 × 224 and has 88 M parameters.…”

“…An additional challenge is that, for a given artist, forgeries are typically much less numerous than original artworks and often lack systematic documentation and high-resolution scans or photos. Despite such limitations, in recent years, NNs such as Convolutional Neural Networks (CNNs) or transformer-based architectures have shown promising results in both art attribution, when trained on datasets of authentic paintings and other stylistically similar artworks [ 17 ] as well as in artwork authentication, trained against forgeries [ 18 ].…”