Multi-Task Deep Learning With Incomplete Training Samples for The Image-Based Prediction of Variables Describing Silk Fabrics

Abstract

<p><strong>Abstract.</strong> This paper presents a method for the classification of images of silk fabrics with the aim to predict properties such as the place and time of origin and the production technique. The proposed method was developed in the context of the EU project SILKNOW (<a href="http://silknow.eu/"target="_blank">http://silknow.eu/</a>). In the context of classification, we address the problem of limited as well as not fully labelled data and investigate the connection between the distinct variables. A pre-trained Convolutional Neural Network (CNN) is used for the feature extraction and a classification network realizing Multi-task learning (MTL) is trained based on these features. The training procedure is adapted to enable the consideration of images that do not have a label for all tasks. Additionally, MTL with fully labeled training data is investigated for the classification of silk fabrics. The impact of both MTL approaches is compared to singletask learning based on two different class structures. We achieve overall accuracies of 92–95 % and average F1-scores of 88–90 % in our best experiments. </p>

From Silk to Digital Technologies: A Gateway to New Opportunities for Creative Industries, Traditional Crafts and Designers. The SILKNOW Case

“…As explained in Section 3.3.2, we are now dealing with an expanded set of semantic properties as well as with images from various collections. Preliminary experiments show that overall accuracies of 60-88% can be achieved, which is considerably worse than the results achieved in [80]. A possible explanation for this is the expanded set of properties, which also leads to a more complex class structure for each property.…”

“…In this scenario, we assessed the impact of considering only complete training samples (MTL-C) as well as considering complete and incomplete samples (MTL-I). Our experiments, the details of which are presented in [80], have shown that the multi-task learning approach leads to a better classification performance compared to the single-task learning approach, but only when exclusively complete samples are used. The results of this approach are discussed in Section 4.2.1.…”

“…In this part, we will discuss the multi-task classification performance of the image classifier presented in Section 3.3.2. In our research from [80], we mainly considered three different scenarios, namely single-task learning (STL), multi-task learning with complete samples (MTL-C) and multi-task learning with complete and incomplete samples (MTL-I). The three tasks, i.e., semantic properties, that were considered are production place, production timespan and production technique.…”

Deep learning for geometric and semantic tasks in photogrammetry and remote sensing

“…describing the time or place of production of a fabric, is very important for art historians, but it is not provided in a standardized way, and sometimes important pieces of information are missing. In the context of an EU H2020 project (SILKNOW 2019), a multi-task CNN based on ResNet (He et al 2015) was developed that simultaneously predicts the production time, the production place and the production technique from a digital image, deriving the training data automatically by analyzing existing collections (Dorozynski, Clermont, and Rottensteiner 2019). The results show that by combining these prediction tasks, the accuracy of prediction is increased if high-quality training samples are used.…”

Assessing the Semantic Similarity of Images of Silk Fabrics Using Convolutional Neural Networks

“…The annotations for these properties are incomplete, so that there is a considerable number of incomplete samples. The dataset used in this paper is identical to the one used by Dorozynski et al (2019). It was generated automatically from the online collection (IMATEX, 2018); in this process, the raw annotations were mapped to a standardized class structure.…”