Deep learning approaches to pattern extraction and recognition in paintings and drawings: an overview

Castellano, Giovanna; Vessio, Gennaro

doi:10.1007/s00521-021-05893-z

Cited by 59 publications

(33 citation statements)

References 84 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The performed analyses span several classification tasks and techniques: from style classification to artist identification, comprising also medium, school, and year classification [ 27 , 28 , 29 ]. These researches are useful to support cultural heritage studies and asset management, e.g., automatic cataloguing of unlabeled works in online and museum collections, but their results can be exploited for more complex applications, such as authentication, stylometry [ 30 ], and forgery detection [ 31 ].…”

Section: Related Workmentioning

confidence: 99%

Comparing CAM Algorithms for the Identification of Salient Image Features in Iconography Artwork Analysis

et al. 2021

View full text Add to dashboard Cite

Iconography studies the visual content of artworks by considering the themes portrayed in them and their representation. Computer Vision has been used to identify iconographic subjects in paintings and Convolutional Neural Networks enabled the effective classification of characters in Christian art paintings. However, it still has to be demonstrated if the classification results obtained by CNNs rely on the same iconographic properties that human experts exploit when studying iconography and if the architecture of a classifier trained on whole artwork images can be exploited to support the much harder task of object detection. A suitable approach for exposing the process of classification by neural models relies on Class Activation Maps, which emphasize the areas of an image contributing the most to the classification. This work compares state-of-the-art algorithms (CAM, Grad-CAM, Grad-CAM++, and Smooth Grad-CAM++) in terms of their capacity of identifying the iconographic attributes that determine the classification of characters in Christian art paintings. Quantitative and qualitative analyses show that Grad-CAM, Grad-CAM++, and Smooth Grad-CAM++ have similar performances while CAM has lower efficacy. Smooth Grad-CAM++ isolates multiple disconnected image regions that identify small iconographic symbols well. Grad-CAM produces wider and more contiguous areas that cover large iconographic symbols better. The salient image areas computed by the CAM algorithms have been used to estimate object-level bounding boxes and a quantitative analysis shows that the boxes estimated with Grad-CAM reach 55% average IoU, 61% GT-known localization and 31% mAP. The obtained results are a step towards the computer-aided study of the variations of iconographic elements positioning and mutual relations in artworks and open the way to the automatic creation of bounding boxes for training detectors of iconographic symbols in Christian art images.

show abstract

Section: Related Workmentioning

confidence: 99%

Comparing CAM Algorithms for the Identification of Salient Image Features in Iconography Artwork Analysis

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Various methods of automatic image segmentation are used in the literature aiming at identifying regions in an image and labeling them as different classes. The main applications are pattern recognition for classifying paintings [19][20][21][22][23] or the authentication of fine arts (e.g., of paintings) [24]. These image segmentation methods include the following: The thresholding methods transform a grey-scale image into a binary image, where the algorithm evaluates the differences among neighboring pixels to find object boundaries [25][26][27].…”

Section: Related Workmentioning

confidence: 99%

“…With the growing volume of information and computing power, neural systems having increasingly sophisticated architecture have been of great interest and are used in a variety of disciplines. Some examples of applications in image processing and in fine arts are as follows: Image segmentation using a neural network has recently been used as a very strong tool for image processing [22,37]; recently, even convolutional neural networks have been applied to paintings [38]. In [39], a novel deep learning framework is developed to retrieve similar architectural floor plan layouts from a repository, analyzing the effect of individual deep convolutional neural network layers for the floor plan retrieval task.…”

Section: Related Workmentioning

confidence: 99%

Classification of Geometric Forms in Mosaics Using Deep Neural Network

et al. 2021

View full text Add to dashboard Cite

The paper addresses an image processing problem in the field of fine arts. In particular, a deep learning-based technique to classify geometric forms of artworks, such as paintings and mosaics, is presented. We proposed and tested a convolutional neural network (CNN)-based framework that autonomously quantifies the feature map and classifies it. Convolution, pooling and dense layers are three distinct categories of levels that generate attributes from the dataset images by introducing certain specified filters. As a case study, a Roman mosaic is considered, which is digitally reconstructed by close-range photogrammetry based on standard photos. During the digital transformation from a 2D perspective view of the mosaic into an orthophoto, each photo is rectified (i.e., it is an orthogonal projection of the real photo on the plane of the mosaic). Image samples of the geometric forms, e.g., triangles, squares, circles, octagons and leaves, even if they are partially deformed, were extracted from both the original and the rectified photos and originated the dataset for testing the CNN-based approach. The proposed method has proved to be robust enough to analyze the mosaic geometric forms, with an accuracy higher than 97%. Furthermore, the performance of the proposed method was compared with standard deep learning frameworks. Due to the promising results, this method can be applied to many other pattern identification problems related to artworks.

show abstract

“…The color noise could not be addressed by the conventional method and it is still a challenging issue in practical applications. Deep learning [ 27 ] has developed rapidly in recent years and has been applied in image processing [ 28 ], language translation [ 29 ], pattern recognition [ 30 ], and other engineering fields [ 31 ]. Since the neural network method could approach the optimal solution, this method could effectively improve the navigation accuracy when the sensor has a large deviation.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Navigation Algorithm with Deep Learning for Autonomous Underwater Vehicle

2021

Sensors

View full text Add to dashboard Cite

Precise navigation is essential for autonomous underwater vehicles (AUVs). The measurement deviation of the navigation sensors, especially the microelectromechanical systems (MEMS) sensors, is a crucial factor that affects the localization accuracy. Deep learning is a novel method to solve this problem. However, the calculation cycle and robustness of the deep learning method may be insufficient in practical application. This paper proposes an adaptive navigation algorithm with deep learning to address these questions and realize accurate navigation. Firstly, this algorithm uses deep learning to generate low-frequency position information to correct the error accumulation of the navigation system. Secondly, the χ2 rule is selected to judge if the Doppler velocity log (DVL) measurement fails, which could avoid interference from DVL outliers. Thirdly, the adaptive filter, based on the variational Bayesian (VB) method, is employed to estimate the navigation information simultaneous with the measurement covariance, improving navigation accuracy even more. The experimental results, based on AUV field data, show that the proposed algorithm could realize robust navigation performance and significantly improve position accuracy.

show abstract

Deep learning approaches to pattern extraction and recognition in paintings and drawings: an overview

Cited by 59 publications

References 84 publications

Comparing CAM Algorithms for the Identification of Salient Image Features in Iconography Artwork Analysis

Comparing CAM Algorithms for the Identification of Salient Image Features in Iconography Artwork Analysis

Classification of Geometric Forms in Mosaics Using Deep Neural Network

Adaptive Navigation Algorithm with Deep Learning for Autonomous Underwater Vehicle

Contact Info

Product

Resources

About