StrucTexT: Structured Text Understanding with Multi-Modal Transformers

Li, Yulin; Qian, Yuxi; Yu, Yuechen; Qin, Xiameng; Zhang, Chengquan; Liu, Yan; Yao, Kun; Han, Junyu; Liu, Jingtuo; Ding, Errui

doi:10.1145/3474085.3475345

Cited by 81 publications

(54 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SelfDoc (Li et al, 2021b) established the contextualization over a block of content, while StructuralLM (Li et al, 2021a) proposed cell-level 2D position embeddings and the corresponding pre-training objective. Recently, StrucTexT (Li et al, 2021c) introduced a unified solution to efficiently extract semantic features from different levels and modalities to handle the entity labeling and entity linking tasks. Doc-Former (Appalaraju et al, 2021) designed a novel multi-modal self-attention layer capable of fusing textual, vision and spatial features.…”

Section: Related Workmentioning

confidence: 99%

LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding

Wang¹,

Jin²,

Ding³

2022

Preprint

View full text Add to dashboard Cite

Structured document understanding has attracted considerable attention and made significant progress recently, owing to its crucial role in intelligent document processing. However, most existing related models can only deal with the document data of specific language(s) (typically English) included in the pre-training collection, which is extremely limited. To address this issue, we propose a simple yet effective Language-independent Layout Transformer (LiLT) for structured document understanding. LiLT can be pretrained on the structured documents of a single language and then directly fine-tuned on other languages with the corresponding offthe-shelf monolingual/multilingual pre-trained textual models. Experimental results on eight languages have shown that LiLT can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which enables language-independent benefit from the pre-training of document layout structure. Code and model are publicly available at https://github.com/jpWang/LiLT.

show abstract

Section: Related Workmentioning

confidence: 99%

LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding

Wang¹,

Jin²,

Ding³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Most previous pre-training models [10], [35] produce an embedding sequence by collecting multimodal information from text, vision, and layout, and then perform a transformer network to establish deep fusion on different modalities. In our work, we adopt semantically meaningful components (e.g., text block, table, figure) as the model input.…”

Section: Gate Fusion Layermentioning

confidence: 99%

Multimodal Pre-training Based on Graph Attention Network for Document Understanding

Zhang¹,

Ma²,

Du³

et al. 2022

Preprint

View full text Add to dashboard Cite

Document intelligence as a relatively new research topic supports many business applications. Its main task is to automatically read, understand, and analyze documents. However, due to the diversity of formats (invoices, reports, forms, etc.) and layouts in documents, it is difficult to make machines understand documents. In this paper, we present the GraphDoc, a multimodal graph attention-based model for various document understanding tasks. GraphDoc is pre-trained in a multimodal framework by utilizing text, layout, and image information simultaneously. In a document, a text block relies heavily on its surrounding contexts, so we inject the graph structure into the attention mechanism to form a graph attention layer so that each input node can only attend to its neighborhoods. The input nodes of each graph attention layer are composed of textual, visual, and positional features from semantically meaningful regions in a document image. We do the multimodal feature fusion of each node by the gate fusion layer. The contextualization between each node is modeled by the graph attention layer. GraphDoc learns a generic representation from only 320k unlabeled documents via the Masked Sentence Modeling task. Extensive experimental results on the publicly available datasets show that GraphDoc achieves state-of-the-art performance, which demonstrates the effectiveness of our proposed method.

show abstract

“…These entity fields are extracted either from preconditioned OCR results or parsing results from electronic formats like PDF or Microsoft Word. To keep a fixed input format, these entities are sorted beforehand according to the top-left to bottom-right order [22].…”

Section: Approachmentioning

confidence: 99%

“…Owing to their huge potential, VRDs understanding has attracted increasing attention in the multimedia community. Recent studies [16,19,22,41,42] usually follow a two-step pre-training then fine-tuning regime. With cutting-edge model designs, the pre-training step jointly models a multi-modal interaction between text, layout, and image, then produces comprehensive representations for fine-tuning specific downstream tasks.…”

Section: Introductionmentioning

confidence: 99%

Relational Representation Learning in Visually-Rich Documents

Li¹,

Zheng²,

Hu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Relational understanding is critical for a number of visually-rich documents (VRDs) understanding tasks. Through multi-modal pre-training, recent studies provide comprehensive contextual representations and exploit them as prior knowledge for downstream tasks. In spite of their impressive results, we observe that the widespread relational hints (e.g., relation of key/value fields on receipts) built upon contextual knowledge are not excavated yet. To mitigate this gap, we propose DocReL, a Document Relational Representation Learning framework. The major challenge of DocReL roots in the variety of relations. From the simplest pairwise relation to the complex global structure, it is infeasible to conduct supervised training due to the definition of relation varies and even conflicts in different tasks. To deal with the unpredictable definition of relations, we propose a novel contrastive learning task named Relational Consistency Modeling (RCM), which harnesses the fact that existing relations should be consistent in differently augmented positive views. RCM provides relational representations which are more compatible to the urgent need of downstream tasks, even without any knowledge about the exact definition of relation. DocReL achieves better performance on a wide variety of VRD relational understanding tasks, including table structure recognition, key information extraction and reading order detection.* Equal contribution. † Contact person.

show abstract

StrucTexT: Structured Text Understanding with Multi-Modal Transformers

Cited by 81 publications

References 45 publications

LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding

LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding

Multimodal Pre-training Based on Graph Attention Network for Document Understanding

Relational Representation Learning in Visually-Rich Documents

Contact Info

Product

Resources

About