BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents

Hong, Teakgyu; Kim, Dong-Hyun; Ji, Mingi; Hwang, Wonseok; Nam, Dae‐Hyun; Park, Sungrae

doi:10.48550/arxiv.2108.04539

Cited by 3 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the fine-tuning step, the representation generated beforehand is utilized for tuning downstream tasks. Typical downstream tasks in VRDs are key information extraction [16,26,46], reading order detection [37] and table structure recognition [27], etc. Despite their objectives varying, they could be considered as relational understanding tasks.…”

Section: Relational Understanding In Vrdsmentioning

confidence: 99%

“…The input is an image with known entity fields. As an entity is a meaningful text sequence with a corresponding layout and image, it could be a sentence to be sorted in reading order detection tasks [37], could be a table cell to be located in table structure recognition tasks [24,25,27], and also could be a key / value field to be identified in key information extraction tasks [16,46]. These entity fields are extracted either from preconditioned OCR results or parsing results from electronic formats like PDF or Microsoft Word.…”

Section: Approachmentioning

confidence: 99%

“…The relational representations provided by DocReL rely on precise entity-level feature extraction. Now mainstream relational understanding tasks like table structure recognition [24,25,27,31,33], key information extraction [16,17,46] and reading order detection [37] are born with entitylevel. To explain, the text-fields acquired from precondi- tioned OCR results or parsing results from electronic formats like PDF or Microsoft Word are directly entity-level, and no post-processing is needed.…”

Section: A7 Feature Extraction Of Entities In Broader Scenariosmentioning

confidence: 99%

“…Visually-Rich Documents (VRDs) understanding aims to automatically analyze and extract key information from scanned / digital-born documents, such as insurance quotes, business emails, and financial receipts. Certain relational understanding tasks already play the important role in building electronic archives and developing office automation, which include table structure recognition [24,25,27,31,33], key information extraction [16,17,46] and reading order detection [37], etc. Owing to their huge potential, VRDs understanding has attracted increasing attention in the multimedia community.…”

Section: Introductionmentioning

confidence: 99%

“…Owing to their huge potential, VRDs understanding has attracted increasing attention in the multimedia community. Recent studies [16,19,22,41,42] usually follow a two-step pre-training then fine-tuning regime. With cutting-edge model designs, the pre-training step jointly models a multi-modal interaction between text, layout, and image, then produces comprehensive representations for fine-tuning specific downstream tasks.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Relational Representation Learning in Visually-Rich Documents

Li¹,

Zheng²,

Hu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Relational understanding is critical for a number of visually-rich documents (VRDs) understanding tasks. Through multi-modal pre-training, recent studies provide comprehensive contextual representations and exploit them as prior knowledge for downstream tasks. In spite of their impressive results, we observe that the widespread relational hints (e.g., relation of key/value fields on receipts) built upon contextual knowledge are not excavated yet. To mitigate this gap, we propose DocReL, a Document Relational Representation Learning framework. The major challenge of DocReL roots in the variety of relations. From the simplest pairwise relation to the complex global structure, it is infeasible to conduct supervised training due to the definition of relation varies and even conflicts in different tasks. To deal with the unpredictable definition of relations, we propose a novel contrastive learning task named Relational Consistency Modeling (RCM), which harnesses the fact that existing relations should be consistent in differently augmented positive views. RCM provides relational representations which are more compatible to the urgent need of downstream tasks, even without any knowledge about the exact definition of relation. DocReL achieves better performance on a wide variety of VRD relational understanding tasks, including table structure recognition, key information extraction and reading order detection.* Equal contribution. † Contact person.

show abstract

Section: Relational Understanding In Vrdsmentioning

confidence: 99%

Section: Approachmentioning

confidence: 99%

Section: A7 Feature Extraction Of Entities In Broader Scenariosmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Relational Representation Learning in Visually-Rich Documents

Li¹,

Zheng²,

Hu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Unimodal and Multimodal Representation Training for Relation Extraction

Cooney¹,

Rachel²,

Liam³

et al. 2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Multimodal integration of text, layout and visual information has achieved SOTA results in visually rich document understanding (VrDU) tasks, including relation extraction (RE). However, despite its importance, evaluation of the relative predictive capacity of these modalities is less prevalent. Here, we demonstrate the value of shared representations for RE tasks by conducting experiments in which each data type is iteratively excluded during training. In addition, text and layout data are evaluated in isolation. While a bimodal text and layout approach performs best (F1 = 0.684), we show that text is the most important single predictor of entity relations. Additionally, layout geometry is highly predictive and may even be a feasible unimodal approach. Despite being less effective, we highlight circumstances where visual information can bolster performance. In total, our results demonstrate the efficacy of training joint representations for RE.

show abstract

DCMAI: A Dynamical Cross-Modal Alignment Interaction Framework for Document Key Information Extraction

Deng¹

2022

Preprint

View full text Add to dashboard Cite

<p>Abstract—Document key information extraction (DKIE) is a challenging task that aims to automatically understand documents in their varied formats and layouts (forms, receipts, etc.). Existing pre-trained methods have shown superior performance on multiple DKIE tasks. However, there are three main drawbacks to DKIE. Firstly, these methods do not consider the ambiguities arising from similar text representations before cross-modal interaction. Secondly, they ignore cross-modal feature alignment before cross-modal interaction. Thirdly, self-attention layers in cross-modal interaction suffer from high memory consumption, which hinders joint representation reasoning from all negative samples. To tackle the above limitations, we present a Dynamical Cross-Modal Alignment Interaction framework(DCMAI). Specifically, (1) to disambiguate similar textual representations, a prior knowledge-guided module is formulated to adaptively mine fine-grained visual information to distinguish similar textual representations, which generates a prior visual knowledge-guided text embedding for each token. (2) A crossover alignment loss is proposed to align cross-modal information, which contributes to improving the interaction between visual and text features before cross-modal interaction. (3) To further reasoning joint representation from a cross-modal encoder and effectively mine cross-modal negative samples, we introduce a hierarchical interaction sampling strategy to mine negative samples, and a contrastive loss is applied to optimize joint representation reasoning. We pre-train the DCMAI framework on a public corpus and fine-tune it on several downstream tasks, such as entity extraction, sequence labeling, and document question answering. The proposed DCMAI approach achieves superior performance on various downstream tasks. Code will be open to the public.</p>

show abstract

BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents

Cited by 3 publications

References 0 publications

Relational Representation Learning in Visually-Rich Documents

Relational Representation Learning in Visually-Rich Documents

Unimodal and Multimodal Representation Training for Relation Extraction

DCMAI: A Dynamical Cross-Modal Alignment Interaction Framework for Document Key Information Extraction

Contact Info

Product

Resources

About