Neural Word Search in Historical Manuscript Collections

Abstract

We address the problem of segmenting and retrieving word images in collections of historical manuscripts given a text query. This is commonly referred to as "word spotting". To this end, we first propose an end-to-end trainable model based on deep neural networks that we dub Ctrl-F-Net. The model simultaneously generates region proposals and embeds them into a word embedding space, wherein a search is performed. We further introduce a simplified version called Ctrl-F-Mini. It is faster with similar performance, though it is limited to more easily segmented manuscripts. We evaluate both models on common benchmark datasets and surpass the previous state of the art. Finally, in collaboration with historians, we employ the Ctrl-F-Net to search within a large manuscript collection of over 100 thousand pages, written across two centuries. With only 11 training pages, we enable large scale data collection in manuscript-based historical research. This results in a speed up of data collection and the number of manuscripts processed by orders of magnitude. Given the time consuming manual work required to study old manuscripts in the humanities, quick and robust tools for word spotting has the potential to revolutionise domains like history, religion and language.

Bootstrapping Weakly Supervised Segmentation-free Word Spotting through HMM-based Alignment

“…We adopt the Ctrl-F-Mini model [2] using the Discrete Cosine Transform of Words (DCToW) embedding [15] as our word spotting model 1 . It is a powerful and relatively light-weight model that achieves state-of-the-art results in segmentation-free word spotting.…”

“…During training, models are evaluated on the validation set every 1000 iterations and the best model is kept. Furthermore, all models are trained using on-the-fly augmentation based on the in-place scheme from [2]. Full-page augmentation was avoided due to the computational overhead of re-extracting region proposals on the fly.…”

“…Word spotting has enjoyed good progress in the last few years, both in segmentation-based and segmentationfree settings [1], [2], [3], [4]. Much headway is due to the adoption of deep convolutional neural networks by the community.…”