LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

Abstract

Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pretrained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose Lay-outLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in imagecentric tasks such as document image classification and document layout analysis. The code and models are publicly available at https://aka.ms/layoutlmv3.

VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classification

“…Therefore, the results reported demonstrate that our proposed approach outperforms all the methods that do not require any supplementary information such as layout information as used in [1,2,8,3,4]. Meanwhile, it achieves competitive results against the methods that include layout information in the pre-training setting.…”

“…The mechanisms used to leverage features from document modalities differ one from another. In [1,5,8,2], the authors propose a joint multimodal approach to model the interaction between textual, visual, and layout information in a unified multimodal pre-training network. Besides, [3] exploit cross-modal learning in the pre-training stage to perform a task-agnostic framework to model information across textual, visual, and layout information modalities without requiring document data annotation.…”

“…Specifically, attention learning has seen increased attention lately in the field of document understanding, imagetext matching, and cross-modal retrieval, aiming at learning the internal relations in a text sentence or in an image. To model the internal relationships among different modalities, we adopt the contextualized attention mechanism from natural language processing (NLP) [20] to improve the location accuracy of a document image region in the vision modality for the desired text sequence in the language modality [8,3,4,5]. Our proposal highlights both the cross-modal co-attention (InterMCA), and internal self-attention (IntraMSA) mechanisms which are integrated in the proposed model, which means that self-attention and co-attention are integrated in the proposed model.…”

“…Therefore, recent research has started to consider how to leverage and incorporate the relations within those different modalities in a unified network to capture latent information for ex-ploring better yet effective multimodal representations. Such systems have shown their effectiveness in improving multimodal representation learning in a pretrainthen-finetune paradigm, where models are first pre-trained with large-scale data and then fine-tuned to each downstream task [1,2,3,4,5,6,7,8].…”

XDoc: Unified Pre-training for Cross-Format Document Understanding

“…Document understanding has undoubtedly been an important research topic as documents play an essential role in message delivery in our daily lives . During the past several years, the flourishing blossom of deep learning has witnessed the rapid development of document understanding in various formats, ranging from plain texts (Devlin et al, 2018;Dong et al, 2019), document texts (Xu et al, , 2021aHuang et al, 2022), and web texts Li et al, 2022a;. Recently, pretraining techniques have been the de facto standard for document understanding, where the model is first pre-trained in a self-supervised manner (e.g.…”