Visual Segmentation for Information Extraction from Heterogeneous Visually Rich Documents

Sarkhel, Ritesh; Nandi, Arnab

doi:10.1145/3299869.3319867

Cited by 19 publications

(8 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These NLP-based approaches typically operate on text sequences and do not incorporate visual and layout information. Later studies [6,16,32] firstly tend to explore layout information to aid entity extraction from VRDs. Post-OCR [13] reconstructs the text sequences based on their bounding boxes.…”

Section: Related Workmentioning

confidence: 99%

“…Post-OCR [13] reconstructs the text sequences based on their bounding boxes. VS2 [32] leverages the heterogeneous layout to perform the extraction in visual logical blocks. A range of other methods [6,16,51] represent a document as a 2D grid with text tokens to obtain the contextual embedding.…”

Section: Related Workmentioning

confidence: 99%

“…Early solutions solve the entity tasks by only operating on plain texts, resulting in a semantic ambiguity. Noticing the rich visual information contained in VRDs, several methods [6,16,26,32] exploit 2D layout information to provides complementation for textual content. Besides, for further improvement, mainstream researches [2,21,24,30,38,48,50] usually employ a shallow fusion of text, image, and layout to capture contextual dependencies.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

StrucTexT: Structured Text Understanding with Multi-Modal Transformers

Qian

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Structured text understanding on Visually Rich Documents (VRDs) is a crucial part of Document Intelligence. Due to the complexity of content and layout in VRDs, structured text understanding has been a challenging task. Most existing studies decoupled this problem into two sub-tasks: entity labeling and entity linking, which require an entire understanding of the context of documents at both token and segment levels. However, little work has been concerned with the solutions that efficiently extract the structured data from different levels. This paper proposes a unified framework named StrucTexT, which is flexible and effective for handling both sub-tasks. Specifically, based on the transformer, we introduce a segment-token aligned encoder to deal with the entity labeling and entity linking tasks at different levels of granularity. Moreover, we design a novel pre-training strategy with three self-supervised tasks to learn a richer representation. StrucTexT uses the existing Masked Visual Language Modeling task and the new Sentence Length Prediction and Paired Boxes Direction tasks to incorporate the multi-modal information across text, image, and layout. We evaluate our method for structured text understanding at segment-level and token-level and show it outperforms the state-of-the-art counterparts with significantly superior performance on the FUNSD, SROIE, and EPHOIE datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

StrucTexT: Structured Text Understanding with Multi-Modal Transformers

Qian

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…how the summaries came to be, the answer still remains illusive. Explaining how a machine-generated summary was constructed, has become a necessity under the newly introduced General Data Privacy Regulation Act (ITGP, 2017), especially for applications in enterprise (Sarkhel and Nandi, 2019;Keymanesh et al, 2020) and biomedical domain (Moradi and Ghadiri, 2018;Sarkhel et al, 2018). Some recent efforts have proposed using interpretable heatmaps (Baan et al, 2019) generated from the attention distribution over an input sequence for interpreting model behaviour.…”

Section: Prototype Summarymentioning

confidence: 99%

Interpretable Multi-headed Attention for Abstractive Summarization at Controllable Lengths

Sarkhel¹,

Keymanesh²,

Nandi³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

Self Cite

View full text Add to dashboard Cite

ive summarization at controllable lengths is a challenging task in natural language processing. It is even more challenging for domains where limited training data is available or scenarios in which the length of the summary is not known beforehand. At the same time, when it comes to trusting machine-generated summaries, explaining how a summary was constructed in human-understandable terms may be critical. We propose Multi-level Summarizer (MLS), a supervised method to construct abstractive summaries of a text document at controllable lengths.The key enabler of our method is an interpretable multi-headed attention mechanism that computes attention distribution over an input document using an array of timestep independent semantic kernels. Each kernel optimizes a human-interpretable syntactic or semantic property. Exhaustive experiments on two low-resource datasets in English language show that MLS outperforms strong baselines by up to 14.70% in the METEOR score. Human evaluation of the summaries also suggests that they capture the key concepts of the document at various length-budgets.

show abstract

“…However, as far as the state of the art is concerned, there are many machine learning models which must be trained for general named entities to be robust (Peters et al, 2018;Akbik et al, 2018;Devlin et al, 2018). To further increase training efficiency, we can use the documents of a previously defined layout, so that the models could learn how to extract a particular piece of information (Zhao et al, 2019;Denk and Reisswig, 2019;Liu et al, 2019a;Sarkhel and Nandi, 2019). On the other hand, more general extractors are still needed to deal with a variety of information.…”

Section: Introductionmentioning

confidence: 99%

Kleister: A novel task for Information Extraction involving Long Documents with Complex Layout

Graliński,

Stanisławek,

Wróblewska

et al. 2020

Preprint

View full text Add to dashboard Cite

State-of-the-art solutions for Natural Language Processing (NLP) are able to capture a broad range of contexts, like the sentence-level context or document-level context for short documents. But these solutions are still struggling when it comes to longer, real-world documents with the information encoded in the spatial structure of the document, such as page elements like tables, forms, headers, openings or footers; complex page layout or presence of multiple pages.To encourage progress on deeper and more complex Information Extraction (IE) we introduce a new task (named Kleister) with two new datasets. Utilizing both textual and structural layout features, an NLP system must find the most important information, about various types of entities, in long formal documents. We propose Pipeline method as a text-only baseline with different Named Entity Recognition architectures (Flair, BERT, RoBERTa). Moreover, we checked the most popular PDF processing tools for text extraction (pdf2djvu, Tesseract and Textract) in order to analyze behavior of IE system in presence of errors introduced by these tools.

show abstract

Visual Segmentation for Information Extraction from Heterogeneous Visually Rich Documents

Cited by 19 publications

References 32 publications

StrucTexT: Structured Text Understanding with Multi-Modal Transformers

StrucTexT: Structured Text Understanding with Multi-Modal Transformers

Interpretable Multi-headed Attention for Abstractive Summarization at Controllable Lengths

Kleister: A novel task for Information Extraction involving Long Documents with Complex Layout

Contact Info

Product

Resources

About