SelfDoc: Self-Supervised Document Representation Learning

Abstract

We propose SelfDoc, a task-agnostic pre-training framework for document image understanding. Because documents are multimodal and are intended for sequential reading, our framework exploits the positional, textual, and visual information of every semantically meaningful component in a document, and it models the contextualization between each block of content. Unlike existing document pre-training models, our model is coarse-grained instead of treating individual words as input, therefore avoiding an overly fine-grained with excessive contextualization. Beyond that, we introduce cross-modal learning in the model pre-training phase to fully leverage multimodal information from unlabeled documents. For downstream usage, we propose a novel modality-adaptive attention mechanism for multimodal feature fusion by adaptively emphasizing language and vision signals. Our framework benefits from self-supervised pre-training on documents without requiring annotations by a feature masking training strategy. It achieves superior performance on multiple downstream tasks with significantly fewer document images used in the pre-training stage compared to previous works.

VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classification

“…Therefore, the results reported demonstrate that our proposed approach outperforms all the methods that do not require any supplementary information such as layout information as used in [1,2,8,3,4]. Meanwhile, it achieves competitive results against the methods that include layout information in the pre-training setting.…”

“…In [1,5,8,2], the authors propose a joint multimodal approach to model the interaction between textual, visual, and layout information in a unified multimodal pre-training network. Besides, [3] exploit cross-modal learning in the pre-training stage to perform a task-agnostic framework to model information across textual, visual, and layout information modalities without requiring document data annotation. In [4], the authors encourage multimodal interaction using a multimodal transformer architecture to perform visual document understanding.…”

“…Specifically, attention learning has seen increased attention lately in the field of document understanding, imagetext matching, and cross-modal retrieval, aiming at learning the internal relations in a text sentence or in an image. To model the internal relationships among different modalities, we adopt the contextualized attention mechanism from natural language processing (NLP) [20] to improve the location accuracy of a document image region in the vision modality for the desired text sequence in the language modality [8,3,4,5]. Our proposal highlights both the cross-modal co-attention (InterMCA), and internal self-attention (IntraMSA) mechanisms which are integrated in the proposed model, which means that self-attention and co-attention are integrated in the proposed model.…”

“…Therefore, recent research has started to consider how to leverage and incorporate the relations within those different modalities in a unified network to capture latent information for ex-ploring better yet effective multimodal representations. Such systems have shown their effectiveness in improving multimodal representation learning in a pretrainthen-finetune paradigm, where models are first pre-trained with large-scale data and then fine-tuned to each downstream task [1,2,3,4,5,6,7,8].…”