No abstract

<p><strong>Abstract.</strong> This paper presents a method for the classification of images of silk fabrics with the aim to predict properties such as the place and time of origin and the production technique. The proposed method was developed in the context of the EU project SILKNOW (<a href="http://silknow.eu/"target="_blank">http://silknow.eu/</a>). In the context of classification, we address the problem of limited as well as not fully labelled data and investigate the connection between the distinct variables. A pre-trained Convolutional Neural Network (CNN) is used for the feature extraction and a classification network realizing Multi-task learning (MTL) is trained based on these features. The training procedure is adapted to enable the consideration of images that do not have a label for all tasks. Additionally, MTL with fully labeled training data is investigated for the classification of silk fabrics. The impact of both MTL approaches is compared to singletask learning based on two different class structures. We achieve overall accuracies of 92&amp;ndash;95&amp;thinsp;% and average F1-scores of 88&amp;ndash;90&amp;thinsp;% in our best experiments. </p>

Nowadays, cultural heritage is more than ever linked to the present. It links us to our cultural past through the conscious act of preserving and bequeathing to future generations, turning society into its custodian. The appreciation of cultural heritage happens not only because of its communicative power, but also because of its economic power, through sustainable development and the promotion of creative industries. This paper presents SILKNOW, an EU-H2002 funded project and its application to cultural heritage, as well as to creative industries and design innovation. To this end, it presents the use of image recognition tools applied to cultural heritage, through the interoperability of data in the open-access registers of silk museums and its presentation, analysis and creative process carried out by the design students of EASD Valencia as a case study, in the branches of jewellery and fashion project, inspired by the heritage of silk.

We develop a multimodal classifier for the cultural heritage domain using a late fusion approach and introduce a novel dataset. The three modalities are Image, Text, and Tabular data. We based the image classifier on a ResNet convolutional neural network architecture and the text classifier on a multilingual transformer architecture (XML-Roberta). Both are trained as multitask classifiers and use the focal loss to handle class imbalance. Tabular data and late fusion are handled by Gradient Tree Boosting. We also show how we leveraged specific data models and taxonomy in a Knowledge Graph to create the dataset and to store classification results. All individual classifiers accurately predict missing properties in the digitized silk artifacts, with the multimodal approach providing the best results.

Abstract. This paper proposes several methods for training a Convolutional Neural Network (CNN) for learning the similarity between images of silk fabrics based on multiple semantic properties of the fabrics. In the context of the EU H2020 project SILKNOW (http://silknow.eu/), two variants of training were developed, one based on a Siamese CNN and one based on a triplet architecture. We propose different definitions of similarity and different loss functions for both training strategies, some of them also allowing the use of incomplete information about the training data. We assess the quality of the trained model by using the learned image features in a k-NN classification. We achieve overall accuracies of 93–95% and average F1-scores of 87–92%.