Writer identification and writer retrieval based on NetVLAD with Re‐ranking

Abstract

The issue of writer identification and writer retrieval, which is considered a challenging problem in the field of document analysis and recognition is addressed here.. A novel pipeline is proposed for the problem at hand by employing a unified neural network architecture consisting of the ResNet-20 as a feature extractor and an integrated NetV-LAD layer, inspired by the vector of locally aggregated descriptors (VLAD), in the head of the latter part. Having defined this architecture, the triplet semi-hard loss function is used to directly learn an embedding for individual input image patches. Subsequently, the generalised max-pooling technique is employed for the aggregation of embedded descriptors of each handwritten image. Also, a novel re-ranking strategy is introduced for the task of identification and retrieval based on the k-reciprocal nearest neighbours, and it is shown that the pipeline can benefit tremendously from this step. Experimental evaluation has been done on the three publicly available datasets: the ICDAR 2013, CVL, and KHATT datasets. Results indicate that while the performance was comparable to the state-of-the-art KHATT, the writer identification and writer retrieval pipeline achieve superior performance on the ICDAR 2013 and CVL datasets in terms of mAP.This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.

Self-supervised Vision Transformers with Data Augmentation Strategies Using Morphological Operations for Writer Retrieval

“…Similarly, CNNs are applied in [6,14] as a feature extractor followed by VLAD. The authors of [20,22] investigate NetVLAD [1], a learnable version of VLAD, plugged in at the end of the network to directly learn the codebook during training. All of those networks are trained in a supervised manner with the writer label as target.…”

“…While reranking is a method to improve the performance of image retrieval in general, two approaches [13,22] investigate reranking in the domain of writer retrieval.…”

“…In [22], Rasoulzadeh and Babaali propose an adaption to the standard reranking method Query Expansion (QE) [8]. They average each descriptor with their top k Reciprocal Nearest Neighbor (kRNN) and show that they can boost the retrieval performance by reducing the effect of false matches.…”

“…State-of-the-art methods for writer retrieval consist of four parts: First, characteristics of the handwriting within the document are sampled, e.g., by using interest point detectors such as SIFT [14,20,22]. Then, traditional algorithms or deep-learning-based approaches are applied to extract features.…”

“…Since the datasets contain a training and a test set with disjunct writers, the performance of writer retrieval approaches is evaluated by using each document of the test set as a query once. While on modern datasets, neural networks trained in a supervised manner dominate [10,14,20,22], for historical datasets, training on writer label information [19,25] trails either unsupervised methods [5] or approaches based on handcrafted features [16]. Historical data introduces additional challenges, e.g., degradation, different languages, the amount of text, or even potential writerlabel noise by external influences on handwriting, such as the pen used.…”