ICDAR 2021 Competition on Document Visual Question Answering

Tito, Rubèn; Mathew, Minesh; Jawahar, C. V.; Valveny, Ernest; Karatzas, Dìmosthenis

doi:10.1007/978-3-030-86337-1_42

Cited by 20 publications

(13 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…BROS was proposed with an effective pretraining method (i.e., area masking) and a relative positional encoding trick. To validate the effectiveness of Webvicob-generated data, we pretrain BROS and measure performance on DocVQA Task 1 (Tito et al, 2021) and Task 3 (Mathew et al, 2022).…”

Section: Comparison Methodsmentioning

confidence: 99%

Technical Report on Web-based Visual Corpus Construction for Visual Document Understanding

Kim¹,

Hong²,

Yim³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a dataset generator engine named Web-based Visual Corpus Builder (Webvicob). Webvicob can readily construct a large-scale visual corpus (i.e., images with text annotations) from a raw Wikipedia HTML dump. In this report, we validate that Webvicobgenerated data can cover a wide range of context and knowledge and helps practitioners to build a powerful Visual Document Understanding (VDU) backbone. The proposed engine is publicly available at https:// github.com/clovaai/webvicob.

show abstract

Section: Comparison Methodsmentioning

confidence: 99%

Technical Report on Web-based Visual Corpus Construction for Visual Document Understanding

Kim¹,

Hong²,

Yim³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, these approaches are unsuitable for real-world applications due to their high computational cost. For example, answering questions in DocVQA (Tito et al, 2021) requires an average of nearly 400 and a maximum of 4,000 OCR tokens (See Section 4.3 and Figure 7). By incorporating the learned queries mechanism and considering the application context, the integration of Cream and LLMs becomes more flexible, enabling the LLM to focus on specific aspects of visual input while generating accurate and contextually appropriate responses.…”

Section: Integration Of Cream and Llmsmentioning

confidence: 99%

“…Our models are evaluated using the test sets of ChartQA (Masry et al, 2022), Infograph-icVQA (Mathew et al, 2022), and DocVQA (Tito et al, 2021), in order to gauge their effectiveness in accurately answering natural language queries reliant on a profound understanding and recognition of various image elements, such as text, objects, and relationships. A sample of the test datasets used is depicted in Figure 6.…”

Section: Test Datasetsmentioning

confidence: 99%

Three-dimensional electronic microfliers inspired by wind-dispersed seeds

Kim

et al. 2021

Nature

145

103

View full text Add to dashboard Cite

Advances in Large Language Models (LLMs) have inspired a surge of research exploring their expansion into the visual domain. While recent models exhibit promise in generating abstract captions for images and conducting natural conversations, their performance on textrich images leaves room for improvement. In this paper, we propose the Contrastive Reading Model (Cream), a novel neural architecture designed to enhance the language-image understanding capability of LLMs by capturing intricate details typically overlooked by existing methods. Cream integrates vision and auxiliary encoders, complemented by a contrastive feature alignment technique, resulting in a more effective understanding of textual information within document images. Our approach, thus, seeks to bridge the gap between vision and language understanding, paving the way for more sophisticated Document Intelligence Assistants. Rigorous evaluations across diverse tasks, such as visual question answering on document images, demonstrate the efficacy of Cream as a state-of-the-art model in the field of visual document understanding. We provide our codebase and newly-generated datasets at https://github.com/naver-ai/cream.

show abstract

“…Document Intelligence can be considered as an umbrella term covering problems of Key Information Extraction [10,54], Table Detection [41,38] and Structure Recognition [39,55], Document Layout Segmentation [5,4] Document Layout Generation [6,36,3,48], Document Visual Question Answering [51,50,32], Document Image Enhancement [49,22,47] which involves the understanding of visually rich semantic information and structure of different layout entities of a whole page.…”

Section: Related Workmentioning

confidence: 99%

“…The Ryerson Vision Lab Complex Document Information Processing (RVL-CDIP) [18] dataset used the IIT-CDIP metadata to create a new dataset for document classification. PublayNet [56] and DocBank [26] are datasets designed for layout analysis tasks and DocVQA [33,51] instead, is designed for Visual Question Answering task over document images.…”

Section: Comparison To Existing Datasetsmentioning

confidence: 99%

OCR-IDL: OCR Annotations for Industry Document Library Dataset

Biten¹,

Tito²,

Gómez³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Pretraining has proven successful in Document Intelligence tasks where deluge of documents are used to pretrain the models only later to be finetuned on downstream tasks. One of the problems of the pretraining approaches is the inconsistent usage of pretraining data with different OCR engines leading to incomparable results between models. In other words, it is not obvious whether the performance gain is coming from diverse usage of amount of data and distinct OCR engines or from the proposed models. To remedy the problem, we make public the OCR annotations for IDL documents using commercial OCR engine given their superior performance over open source OCR models. The contributed dataset (OCR-IDL) has an estimated monetary value over 20K US$. It is our hope that OCR-IDL can be a starting point for future works on Document Intelligence. All of our data and its collection process with the annotations can be found in https://github.com/furkanbiten/idl_data.

show abstract

ICDAR 2021 Competition on Document Visual Question Answering

Cited by 20 publications

References 23 publications

Technical Report on Web-based Visual Corpus Construction for Visual Document Understanding

Technical Report on Web-based Visual Corpus Construction for Visual Document Understanding

Three-dimensional electronic microfliers inspired by wind-dispersed seeds

OCR-IDL: OCR Annotations for Industry Document Library Dataset

Contact Info

Product

Resources

About