Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learning

Pramanik, Subhojeet; Mujumdar, Shashank; Patel, Hima

doi:10.48550/arxiv.2009.14457

Cited by 3 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These include two standard convolutional neural networks VGG-16 and ResNet-50, two multimodal ensemble approaches [13,8] using VGG-16 and a neural network for text encoding, plus the Sentence-BERT embedding. We have four task-agnostic learning methods, including two pre-trained language models [9,19], the approach proposed by Pramanik et al [25] pre-trained on arXiv dataset [4], and LayoutLM [36] pre-trained on IIT-CDIP dataset [17]. SelfDoc outperforms baselines.…”

Section: Resultsmentioning

confidence: 99%

“…Document Pre-training. Most recently, some works have started pre-training models on document images [36,25]. The first one, LayoutLM [36], inherits the main idea from BERT while receiving the extra positional information for text in documents, and additionally includes image embeddings in the fine-tuning phase.…”

Section: Related Workmentioning

confidence: 99%

“…The first one, LayoutLM [36], inherits the main idea from BERT while receiving the extra positional information for text in documents, and additionally includes image embeddings in the fine-tuning phase. Pramanik et al [25] use Longformer [6] for heavily-word documents and extend the pre-training strategies to multi-page document pre-training. In [7], they introduce a document pre-training method by solving jigsaw puzzles and doing multimodal learning via topic modeling.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

SelfDoc: Self-Supervised Document Representation Learning

Li¹,

Gu²,

Kuen³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose SelfDoc, a task-agnostic pre-training framework for document image understanding. Because documents are multimodal and are intended for sequential reading, our framework exploits the positional, textual, and visual information of every semantically meaningful component in a document, and it models the contextualization between each block of content. Unlike existing document pre-training models, our model is coarse-grained instead of treating individual words as input, therefore avoiding an overly fine-grained with excessive contextualization. Beyond that, we introduce cross-modal learning in the model pre-training phase to fully leverage multimodal information from unlabeled documents. For downstream usage, we propose a novel modality-adaptive attention mechanism for multimodal feature fusion by adaptively emphasizing language and vision signals. Our framework benefits from self-supervised pre-training on documents without requiring annotations by a feature masking training strategy. It achieves superior performance on multiple downstream tasks with significantly fewer document images used in the pre-training stage compared to previous works.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

SelfDoc: Self-Supervised Document Representation Learning

Li¹,

Gu²,

Kuen³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, pre-training models [7,22,35,36] show a strong feature representation using large-scale unlabeled training samples. Inspired by this, several works [28,45,46] combine pretraining techniques to improve multi-modal features. Pramanik et al [28] introduces a multi-task learning-based framework to yield a generic document representation.…”

Section: Related Workmentioning

confidence: 99%

“…Besides, for further improvement, mainstream researches [2,21,24,30,38,48,50] usually employ a shallow fusion of text, image, and layout to capture contextual dependencies. Recently, several pre-training models [28,45,46] have been proposed for joint learning the deep fusion of cross-modality on large-scale data and outperform counterparts on document understanding. Although these pre-training models consider all modalities of documents, they focus on the contribution related to the text side with less elaborate visual features.…”

Section: Introductionmentioning

confidence: 99%

StrucTexT: Structured Text Understanding with Multi-Modal Transformers

Qian

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Structured text understanding on Visually Rich Documents (VRDs) is a crucial part of Document Intelligence. Due to the complexity of content and layout in VRDs, structured text understanding has been a challenging task. Most existing studies decoupled this problem into two sub-tasks: entity labeling and entity linking, which require an entire understanding of the context of documents at both token and segment levels. However, little work has been concerned with the solutions that efficiently extract the structured data from different levels. This paper proposes a unified framework named StrucTexT, which is flexible and effective for handling both sub-tasks. Specifically, based on the transformer, we introduce a segment-token aligned encoder to deal with the entity labeling and entity linking tasks at different levels of granularity. Moreover, we design a novel pre-training strategy with three self-supervised tasks to learn a richer representation. StrucTexT uses the existing Masked Visual Language Modeling task and the new Sentence Length Prediction and Paired Boxes Direction tasks to incorporate the multi-modal information across text, image, and layout. We evaluate our method for structured text understanding at segment-level and token-level and show it outperforms the state-of-the-art counterparts with significantly superior performance on the FUNSD, SROIE, and EPHOIE datasets.

show abstract

Unimodal and Multimodal Representation Training for Relation Extraction

Cooney¹,

Rachel²,

Liam³

et al. 2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Multimodal integration of text, layout and visual information has achieved SOTA results in visually rich document understanding (VrDU) tasks, including relation extraction (RE). However, despite its importance, evaluation of the relative predictive capacity of these modalities is less prevalent. Here, we demonstrate the value of shared representations for RE tasks by conducting experiments in which each data type is iteratively excluded during training. In addition, text and layout data are evaluated in isolation. While a bimodal text and layout approach performs best (F1 = 0.684), we show that text is the most important single predictor of entity relations. Additionally, layout geometry is highly predictive and may even be a feasible unimodal approach. Despite being less effective, we highlight circumstances where visual information can bolster performance. In total, our results demonstrate the efficacy of training joint representations for RE.

show abstract

Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learning

Cited by 3 publications

References 19 publications

SelfDoc: Self-Supervised Document Representation Learning

SelfDoc: Self-Supervised Document Representation Learning

StrucTexT: Structured Text Understanding with Multi-Modal Transformers

Unimodal and Multimodal Representation Training for Relation Extraction

Contact Info

Product

Resources

About