One-shot Text Field labeling using Attention and Belief Propagation for Structure Information Extraction

Abstract

Structured information extraction from document images usually consists of three steps: text detection, text recognition, and text field labeling. While text detection and text recognition have been heavily studied and improved a lot in literature, text field labeling is less explored and still faces many challenges. Existing learning based methods for text labeling task usually require a large amount of labeled examples to train a specific model for each type of document. However, collecting large amounts of document images and labeling them is difficult and sometimes impossible due to privacy issues. Deploying separate models for each type of document also consumes a lot of resources. Facing these challenges, we explore one-shot learning for the text field labeling task. Existing one-shot learning methods for the task are mostly rule-based and have difficulty in labeling fields in crowded regions with few landmarks and fields consisting of multiple separate text regions. To alleviate these problems, we proposed a novel deep end-to-end trainable approach for one-shot text field labeling, which makes use of attention mechanism to transfer the layout information between document images. We further applied conditional random field on the transferred layout information for the refinement of field labeling. We collected and annotated a real-world one-shot field labeling dataset with a large variety of document types and conducted extensive experiments to examine the effectiveness of the proposed model. To stimulate research in this direction, the collected dataset and the one-shot model will be released 1. CCS CONCEPTS • Applied computing → Document analysis; Optical character recognition; • Computing methodologies → Visual contentbased indexing and retrieval.

One-shot Key Information Extraction from Document with Deep Partial Graph Matching

“…Cheng et al [1] presented a one-shot field labeling method using attention and belief propagation to retrieve structured information. Although their method dramatically simplifies the labeling process and achieved good performance compared with previous one-shot-based approaches, the final matching results are not globally optimal.…”

“…In the first subsection, we define all the notations about graphs. In the second subsection, we discuss how to formulate the partial graph matching problem and how to annotate training data to avoid the many-to-many mapping, which violates the definition of graph matching problem [1], between fields. In the third subsection, we report important details of constructing graphs that consists of fields.…”

“…They are supposed to have the same label but are bounded by separate boxes. Cheng et al [1] suggests using the average boxes of support fields that share the same label to match with query fields. As shown in a1 and a2, their method leads to a one-to-many mapping that violates the one-to-(at most)one mapping constraint.…”

“…This process can be time-consuming and tedious. To automate this process, people studied the Key Information Extraction (KIE) task [1]- [3]. Thus, KIE is crucial to a company in terms of efficiency and productivity, and it has been successfully used in many industrial scenarios, such as fast indexing and efficient archiving.…”

“…However, these rule-based methods are limited to specific layouts and are not general enough to scale to all types of documents. Cheng et al [1] proposed an attention-based learning approach to transfer the spatial relationships between landmarks and fields from a support document to a query document. However, their method cannot deal with drifted fields and outliers.…”