Asking questions on handwritten document collections

Abstract

This work addresses the problem of Question Answering (QA) on handwritten document collections. Unlike typical QA and Visual Question Answering (VQA) formulations where the answer is a short text, we aim to locate a document snippet where the answer lies. The proposed approach works without recognizing the text in the documents. We argue that the recognitionfree approach is suitable for handwritten documents and historical collections where robust text recognition is often difficult. At the same time, for human users, document image snippets containing answers act as a valid alternative to textual answers. The proposed approach uses an off-the-shelf deep embedding network which can project both textual words and word images into a common sub-space. This embedding bridges the textual and visual domains and helps us retrieve document snippets that potentially answer a question. We evaluate results of the proposed approach on two new datasets: (i) HW-SQuAD: a synthetic, handwritten document image counterpart of SQuAD1.0 dataset and (ii) BenthamQA: a smaller set of QA pairs defined on documents from the popular Bentham manuscripts collection. We also present a thorough analysis of the proposed recognition-free approach compared to a recognition-based approach which uses text recognized from the images using an OCR. Datasets presented in this work are available to download at docvqa.org.

Self-Training of Handwritten Word Recognition for Synthetic-to-Real Adaptation

“…Interestingly, the approach also leads to considerable performance gains even when the initial model is comparably poor, as shown in the experiments on BT. This further encourages the consideration of adaptation strategies as considerable improvements can be achieved when compared to models solely trained on related data, as presented in [12].…”

“…For GW, IAM and CVL, we follow the exact same evaluation protocols as described in [14]. For BT we follow [12], as they also report recognition performances not relying on any in-domain samples.…”

“…The limited number of samples is especially challenging, when confronted with document collections that have a distinctive style. Considering, for example, historic documents, the appearance is often too specific to simply employ a HTR system trained on modern handwriting [12]. Collection specific annotations need to be created, which often can only be done by experts making it a costly approach.…”