Comparing CAM Algorithms for the Identification of Salient Image Features in Iconography Artwork Analysis

Abstract

Iconography studies the visual content of artworks by considering the themes portrayed in them and their representation. Computer Vision has been used to identify iconographic subjects in paintings and Convolutional Neural Networks enabled the effective classification of characters in Christian art paintings. However, it still has to be demonstrated if the classification results obtained by CNNs rely on the same iconographic properties that human experts exploit when studying iconography and if the architecture of a classifier trained on whole artwork images can be exploited to support the much harder task of object detection. A suitable approach for exposing the process of classification by neural models relies on Class Activation Maps, which emphasize the areas of an image contributing the most to the classification. This work compares state-of-the-art algorithms (CAM, Grad-CAM, Grad-CAM++, and Smooth Grad-CAM++) in terms of their capacity of identifying the iconographic attributes that determine the classification of characters in Christian art paintings. Quantitative and qualitative analyses show that Grad-CAM, Grad-CAM++, and Smooth Grad-CAM++ have similar performances while CAM has lower efficacy. Smooth Grad-CAM++ isolates multiple disconnected image regions that identify small iconographic symbols well. Grad-CAM produces wider and more contiguous areas that cover large iconographic symbols better. The salient image areas computed by the CAM algorithms have been used to estimate object-level bounding boxes and a quantitative analysis shows that the boxes estimated with Grad-CAM reach 55% average IoU, 61% GT-known localization and 31% mAP. The obtained results are a step towards the computer-aided study of the variations of iconographic elements positioning and mutual relations in artworks and open the way to the automatic creation of bounding boxes for training detectors of iconographic symbols in Christian art images.

Proposals Generation for Weakly Supervised Object Detection in Artwork Images

“…To evaluate the proposed pipeline, two artworks’ data sets are used, namely ArtDL 2.0 and IconArt [ 16 ]. The two data sets were selected as representatives of non-natural images for the WSOD task, which was demonstrated to be quite different from WSOD on natural images [ 16 , 17 , 18 ]. They also provide diversity in the image quality and annotated classes.…”

“…For the implementation details, we refer the reader to the original works. A comparison of the localization abilities of CAMs on artworks has been presented in [ 17 ].…”

“…The proposed pipeline consists of: an existing ResNet-50 classifier, a CAM method paired with a dynamic thresholding technique to generate pseudo-ground truth (pseudo-GT) bounding boxes, and a Faster R-CNN object detector to localize classes inside images. We evaluate performances on two artwork data sets (ArtDL 2.0 and IconArt [ 16 ]), which are annotated for WSOD and whose complexity has been demonstrated in previous studies [ 17 , 18 ]. Our approach is able to reach ≈41.5% mAP on ArtDL and ≈17% mAP on IconArt, where state-of-the-art techniques obtain a maximum value of ≈25% and ≈15% mAP, respectively.…”

“…We evaluate performances on two artwork data sets (ArtDL 2.0 and IconArt [ 16 ]), which are annotated for WSOD and whose complexity has been demonstrated in previous studies [ 17 , 18 ]. Our approach is able to reach ≈41.5% mAP on ArtDL and ≈17% mAP on IconArt, where state-of-the-art techniques obtain a maximum value of ≈25% and ≈15% mAP, respectively.…”

Editorial for Special Issue “Fine Art Pattern Extraction and Recognition”

“…To this end, computer vision techniques are good candidates to aid in the automatic categorization and retrieval of artworks. Following this line of research, Pinciroli Vago et al [7] experimentally compare several Class Activation Map techniques, which emphasize the areas of an image that contribute the most to the final classification performed by a convolutional neural network. This effort represents a step towards the creation of a computerized tool that is capable of highlighting variations in the positioning of iconographic elements, particularly for the detection of iconographic symbols in art images.…”