LayoutLM: Pre-training of Text and Layout for Document Image Understanding

Abstract

Automatic information extraction from identity documents is a fundamental task in digital processes such as onboarding, requesting products, identity validation, among others. The information extraction process consists of identifying, locating, classifying and recognizing text of the corresponding key fields that an identity document contains. In the case of identity documents, key fields are: names, last names, document number, dates, among others.The information extraction problem has been traditionally solved using rule based algorithms and classic OCR engines. In the last few years there have been implementations based on machine learning models, using NLP (natural language processing) and CV (computer vision) to solve the problem in a more flexible and efficient way (Subramani et al., 2020). This work proposes to solve the problem of information extraction with an object detection approach. An object detection model based on transformers (Carion et al., 2020) was implemented, trained and evaluated. A solution with above 95% accuracy in detecting key fields on identification documents was achieved.

A Survey of Graphical Page Object Detection with Deep Neural Networks

“…The second promising direction is the multimodal processing of the graphical objects. In the case of graphical page object detection, multimodal processing, in the simplest form, is the processing of image information and text information together [62,63]. An example of such a case is when a figure is categorized as a table and vice versa; the text information can be beneficial.…”

Form2Seq : A Framework for Higher-Order Form Structure Extraction

“…Extracting pre-defined and commonly occurring named entities from invoices like documents(using text and box coordinates) has been the main focus for some prior works (Katti et al, 2018;Liu et al, 2019;Denk and Reisswig, 2019;Majumder et al, 2020). Text and document layouts have been used for learning BERT (Devlin et al, 2019) like representations through pre-training and then combined with image features for information extraction from documents (Xu et al, 2020;Garncarek et al, 2020). However, our work focuses on extracting a much more generic, diverse, complex, dense, and hierarchical document structure from Forms.…”

Layout-Aware Text Representations Harm Clustering Documents by Type

“…LayoutLM (Xu et al, 2019) is a BERT-like transformer model modified to generate layoutaware contextualized word embeddings. In place of BERT's single positional embedding, LayoutLM adds positional embeddings for the x-and ycoordinates of a bounding box around the token.…”

“…We therefore expect that a hybrid document representation that combines layout and text information should outperform a text-only representation when clustering documents by type. LayoutLM (Xu et al, 2019) is such a hybrid system and achieves state-of-theart performance for document-type classification, outperforming text-only baselines. We therefore hypothesized that LayoutLM would also outperform these baselines for document-type clustering.…”