Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings

Efthymiou, Athanasios; Rudinac, Stevan; Kackovic, Monika; Worring, Marcel; Wijnberg, Nachoem M.

doi:10.1145/3474085.3475586

Cited by 9 publications

(4 citation statements)

References 52 publications

(53 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their method relies on generating synthetic images generated with Stable Diffusion and computing CLIP [24] image embeddings to obtain content and style embeddings of paintings in content and style spaces. Efthymiou et al [8] proposed the multimodal architecture, which consists of Graph Neural Networks and Convolutional Neural Networks. These networks were jointly trained on visual and semantic artistic representations.…”

Section: Art Retrievalmentioning

confidence: 99%

Learning to rank approach for refining image retrieval in visual arts

Yemelianenko,

Tkachenko,

Masclef

et al. 2023

2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

Modern content-based image retrieval systems demonstrate rather good performance in identifying visually similar artworks. However, this task becomes more challenging when art history specialists aim to refine the list of similar artworks based on their criteria, thus we need to train the model to reproduce this refinement. In this paper, we propose an approach for improving the list of similar paintings according to specific simulated criteria. By this approach, we retrieve paintings similar to a request image using ResNet50 model and ANNOY algorithm. Then, we simulate re-ranking based on the two criteria, and use the re-ranked lists for training LambdaMART model. Finally, we demonstrate that the trained model reproduces the re-ranking for the query painting by the specific criteria. We plan to use the proposed approach for reproducing re-rankings made by art history specialists, when this data will be collected.

show abstract

Section: Art Retrievalmentioning

confidence: 99%

Learning to rank approach for refining image retrieval in visual arts

Yemelianenko,

Tkachenko,

Masclef

et al. 2023

2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

show abstract

“…The usability of feature vectors learned in the context of image classification to serve as descriptors for image retrieval has already been investigated [28,[30][31][32]. Even leveraging the softmax layer activations for image retrieval seems to be possible [33].…”

Section: Auxiliary Lossesmentioning

confidence: 99%

Deep Descriptor Learning with Auxiliary Classification Loss for Retrieving Images of Silk Fabrics in the Context of Preserving European Silk Heritage

Dorozynski

Rottensteiner

2022

IJGI

View full text Add to dashboard Cite

With the growing number of digitally available collections consisting of images depicting relevant objects from the past in relation with descriptive annotations, the need for suitable information retrieval techniques is becoming increasingly important to support historians in their work. In this context, we address the problem of image retrieval for searching records in a database of silk fabrics. The descriptors, used as an index to the database, are learned by a convolutional neural network, exploiting the available annotations to automatically generate training data. Descriptor learning is combined with auxiliary classification loss with the aim of supporting the clustering in the descriptor space with respect to the properties of the depicted silk objects, such as the place or time of origin. We evaluate our approach on a dataset of fabric images in a kNN-classification, showing promising results with respect to the ability of the descriptors to represent semantic properties of silk fabrics; integrating the auxiliary loss improves the overall accuracy by 2.7% and the average F1 score by 5.6%. It can be observed that the largest improvements can be obtained for variables with imbalanced class distributions. An evaluation on the WikiArt dataset demonstrates the transferability of our approach to other digital collections.

show abstract

“…In this sense, we find revealing and academically documented works such as Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings (Efthymiou et al, 2021), an example of multimodal architecture that integrates Graph Neural Networks (GNNs) and Convolutional Neural Networks (CNNs), to weave a framework of meanings between visual and semantic artistic…”

Section: Introductionmentioning

confidence: 98%

Cartographies and Limits through the accumulation of Imaginaries

Navarro¹

2023

uou

View full text Add to dashboard Cite

Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings

Cited by 9 publications

References 52 publications

Learning to rank approach for refining image retrieval in visual arts

Learning to rank approach for refining image retrieval in visual arts

Deep Descriptor Learning with Auxiliary Classification Loss for Retrieving Images of Silk Fabrics in the Context of Preserving European Silk Heritage

Cartographies and Limits through the accumulation of Imaginaries

Contact Info

Product

Resources

About