How to Read Paintings: Semantic Art Understanding with Multi-modal Retrieval

Abstract

Automatic art analysis has been mostly focused on classifying artworks into different artistic styles. However, understanding an artistic representation involves more complex processes, such as identifying the elements in the scene or recognizing author influences. We present SemArt, a multi-modal dataset for semantic art understanding. SemArt is a collection of fine-art painting images in which each image is associated to a number of attributes and a textual artistic comment, such as those that appear in art catalogues or museum collections. To evaluate semantic art understanding, we envisage the Text2Art challenge, a multi-modal retrieval task where relevant paintings are retrieved according to an artistic text, and vice versa. We also propose several models for encoding visual and textual artistic representations into a common semantic space. Our best approach is able to find the correct image within the top 10 ranked images in the 45.5% of the test samples. Moreover, our models show remarkable levels of art understanding when compared against human evaluation.

Context-Aware Embeddings for Automatic Art Analysis

“…The two proposed context-aware embeddings are evaluated on the SemArt dataset [17] in four different art classification tasks and in two cross-modal retrieval tasks. We show that, although none of the proposed models show a superior performance with respect to the other one in all of the evaluated tasks, context-aware embeddings consistently outperform methods based on visual embeddings only.…”

“…For example, [21] proposed to detect authors by analysing their brushwork using wavelet decompositions, [24,41] combined color, edge, or texture features for author, style, and school classification and [5,32] used SIFT features [28] to classify paintings into different attributes. In the last years, deep visual features extracted from CNNs have been repeatedly shown to be very effective in many computer vision tasks, including automatic art analysis [2,17,23,29,30,37,44,45]. At first, deep features were extracted from pre-trained networks and used off-the-shelf for automatic art classification [2,23,37].…”

“…Later, deep visual features were shown to obtain better results when fine-tuned using painting images [8,30,39,44,45]. Alternatively, [10][11][12] explored domain transfer for object and face recognition in paintings, whereas [17] introduced the use of joint visual and textual models to study paintings from a semantic perspective.…”

Linking Art through Human Poses

“…Detection of people in artworks is implemented in [10] and weakly supervised detection of objects in paintings is investigated in [11]. Multimodal retrieval accross visual and textual representations of paintings is discussed by [12]. The demand for artrelated visual recognition algorithms is best demonstrated by the challenge of [13].…”

Recognizing Characters in Art History Using Deep Learning

“…Recent research on the semantic understanding of paintings in art, however, shows that it is possible to apply transfer learning of vision to artworks while training new models. For example, Garcia and Vogiatzis [16] considered the Text2Art challenge where they successfully retrieved the related artwork from the set of test images 45.5% (Top-10) of times. They also showed a technique to store visual and textual information of the same artwork in the same semantic space, thereby making the task of retrieval much easier.…”