Assessing the Semantic Similarity of Images of Silk Fabrics Using Convolutional Neural Networks

Clermont, D.; Dorozynski, Mareike; Wittich, Dennis; Rottensteiner, Franz

doi:10.5194/isprs-annals-v-2-2020-641-2020

Cited by 2 publications

(1 citation statement)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, in the context of image retrieval in databases of works of art, a gradual concept of similarity [13,14] might be more intuitive than a binary one. One option to define such a non-binary concept of similarity can be obtained by measuring the level of similarity of an image pair by the level of agreement of the semantic annotations for multiple variables-a concept we referred to as semantic similarity in [20,21]. In these works, we also considered the problem of missing information: if harvested automatically from online collections of museums, many records in a database containing information about cultural heritage objects will not contain annotations for all variables considered to be relevant for defining similarity.…”

Section: Introductionmentioning

confidence: 99%

Deep Descriptor Learning with Auxiliary Classification Loss for Retrieving Images of Silk Fabrics in the Context of Preserving European Silk Heritage

Dorozynski

Rottensteiner

2022

IJGI

View full text Add to dashboard Cite

With the growing number of digitally available collections consisting of images depicting relevant objects from the past in relation with descriptive annotations, the need for suitable information retrieval techniques is becoming increasingly important to support historians in their work. In this context, we address the problem of image retrieval for searching records in a database of silk fabrics. The descriptors, used as an index to the database, are learned by a convolutional neural network, exploiting the available annotations to automatically generate training data. Descriptor learning is combined with auxiliary classification loss with the aim of supporting the clustering in the descriptor space with respect to the properties of the depicted silk objects, such as the place or time of origin. We evaluate our approach on a dataset of fabric images in a kNN-classification, showing promising results with respect to the ability of the descriptors to represent semantic properties of silk fabrics; integrating the auxiliary loss improves the overall accuracy by 2.7% and the average F1 score by 5.6%. It can be observed that the largest improvements can be obtained for variables with imbalanced class distributions. An evaluation on the WikiArt dataset demonstrates the transferability of our approach to other digital collections.

show abstract

Section: Introductionmentioning

confidence: 99%

Deep Descriptor Learning with Auxiliary Classification Loss for Retrieving Images of Silk Fabrics in the Context of Preserving European Silk Heritage

Dorozynski

Rottensteiner

2022

IJGI

View full text Add to dashboard Cite

show abstract

Computer Vision Tagging the Metropolitan Museum of Art's Collection

Villaespesa

Crider

2021

J. Comput. Cult. Herit.

View full text Add to dashboard Cite

Computer vision algorithms are increasingly being applied to museum collections to identify patterns, colors, and subjects by generating tags for each object image. There are multiple off-the-shelf systems that offer an accessible and rapid way to undertake this process. Based on the highlights of the Metropolitan Museum of Art's collection, this article examines the similarities and differences between the tags generated by three well-known computer vision systems (Google Cloud Vision, Amazon Rekognition, and IBM Watson). The results provide insights into the characteristics of these taxonomies in terms of the volume of tags generated for each object, their diversity, typology, and accuracy. In consequence, this article discusses the need for museums to define their own subject tagging strategy and selection criteria of computer vision tools based on their type of collection and tags needed to complement their metadata.

show abstract

Assessing the Semantic Similarity of Images of Silk Fabrics Using Convolutional Neural Networks

Cited by 2 publications

References 22 publications

Deep Descriptor Learning with Auxiliary Classification Loss for Retrieving Images of Silk Fabrics in the Context of Preserving European Silk Heritage

Deep Descriptor Learning with Auxiliary Classification Loss for Retrieving Images of Silk Fabrics in the Context of Preserving European Silk Heritage

Computer Vision Tagging the Metropolitan Museum of Art's Collection

Contact Info

Product

Resources

About