Cross-Depicted Historical Motif Categorization and Retrieval with Deep Learning

Abstract

In this paper, we tackle the problem of categorizing and identifying cross-depicted historical motifs using recent deep learning techniques, with aim of developing a content-based image retrieval system. As cross-depiction, we understand the problem that the same object can be represented (depicted) in various ways. The objects of interest in this research are watermarks, which are crucial for dating manuscripts. For watermarks, cross-depiction arises due to two reasons: (i) there are many similar representations of the same motif, and (ii) there are several ways of capturing the watermarks, i.e., as the watermarks are not visible on a scan or photograph, the watermarks are typically retrieved via hand tracing, rubbing, or special photographic techniques. This leads to different representations of the same (or similar) objects, making it hard for pattern recognition methods to recognize the watermarks. While this is a simple problem for human experts, computer vision techniques have problems generalizing from the various depiction possibilities. In this paper, we present a study where we use deep neural networks for categorization of watermarks with varying levels of detail. The macro-averaged F1-score on an imbalanced 12 category classification task is 88.3 %, the multi-labelling performance (Jaccard Index) on a 622 label task is 79.5 %. To analyze the usefulness of an image-based system for assisting humanities scholars in cataloguing manuscripts, we also measure the performance of similarity matching on expert-crafted test sets of varying sizes (50 and 1000 watermark samples). A significant outcome is that all relevant results belonging to the same super-class are found by our system (Mean Average Precision of 100%), despite the cross-depicted nature of the motifs. This result has not been achieved in the literature so far.

COVID-19 classification by CCSHNet with deep fusion using transfer learning and discriminant correlation analysis

“…Measures can also be given at an overall level. One is called macro-level, which computes the metric independently for each class and takes the average that gives equal weight to each class (treating all classes equally) [48] . In contrast, the other is micro-level, weighting all samples equally [49] .…”

Delving into the Depths of Image Retrieval Systems in the Light of Deep Learning: A Review

“…Digital image acquisition and capturing tools have been quite popular in many domains. Utilizations of images in different fields are in trend, be it using in clinical science for diseases detection (1) , for diagnosis in radiology (2) , using remote sensing (3) , use case scenario in communication (4) , document scrutiny by retrieving watermarks (5) are just few examples. The huge image repositories generated through this exploration and the constraints posed by conventional text-based database systems, underscoring the need to explore robust solutions for image databases and adopt efficient strategies to facilitate image retrieval.…”