Matching Handwritten Document Images

Abstract

We address the problem of predicting similarity between a pair of handwritten document images written by different individuals. This has applications related to matching and mining in image collections containing handwritten content. A similarity score is computed by detecting patterns of text re-usages between document images irrespective of the minor variations in word morphology, word ordering, layout and paraphrasing of the content. Our method does not depend on an accurate segmentation of words and lines. We formulate the document matching problem as a structured comparison of the word distributions across two document images. To match two word images, we propose a convolutional neural network (cnn) based feature descriptor. Performance of this representation surpasses the state-of-theart on handwritten word spotting. Finally, we demonstrate the applicability of our method on a practical problem of matching handwritten assignments.

HWNet v2: an efficient word image representation for handwritten documents

“…The baseline cnn architecture HWNet considered in this work was first proposed in [37], which first demon- strated the use of such an architecture in learning an efficient word level representation. This work is dedicated entirely to enrich the representation space and learning better in-variances that are common to the handwritten data.…”

“…In [7,48], profile features were combined with the shape based structural features for a partial matching scheme using dtw. Although these features Variable Length Profile+Moments [46] Profile+DFT [40] Slit HoG [79] Local Gradient Histogram [58] HMM SIFT [63] SIFT [1,2,67,70,86] BoWs SIFT [3] Fisher SIFT [4] PHOC Attributes Deep Learning [37] Neural Codes Deep Learning [35,74,75,83] PHOC Attributes Deep Learning [24] Levenshtein Embedding are fast to compute, it is susceptible to noise and common degradation present in documents.…”

“…In the later set of works [75,76] from the same group, PHOCNet was adapted with temporal pooling layer (TPP-PHOCNet) and evaluated under different loss functions and optimization algorithms which further improved the word spotting performance. In [35,36], the features computed from HWNet [37] are embedded into word attribute space by training attribute based svm classifiers and projecting both image and textual attributes to a common subspace. In [83], the authors propose a two stage architecture where a triplet cnn network is trained to reduce the distance between the anchor word image and a similar labeled (positive) word image, while simultaneously increasing the distance between the anchor and negative labeled word image.…”

“…When the data is limited, fine-tuning a pre-trained network has also been demonstrated to be very effective. In the domain of document images these features have shown better performance for word spotting [37,74,76,83], recognition [52], document classification [26], layout analysis [10], etc. In this work, we propose a deep cnn architecture named as HWNet v2, for the task of learning an efficient word level representation for handwritten documents which can handle multiple writers and, is robust to common forms of degradation and noise.…”

Exploring Confidence Measures for Word Spotting in Heterogeneous Datasets

“…, L} with L being the number of layers (only counting layers that have trainable weights). Here, we choose f 2 , f 4 , f 7 , f 10 , f 13 Dataset Train Test Historic Writers GW [11] PHOCNet ID yes 1 Botany [12] PHOCNet ID yes 1 IAM [13] -OD no 657 HWSynth [5] MC -no 0 are then concatenated and fed through a single neuron with sigmoid activation. First, the PHOCNet weights are trained as described in Sec.…”