OCR-Free Document Understanding Transformer

Hong, Teakgyu; Yim, Moonbin; Nam, Jeongyeon; Park, Jin‐Young; Yim, Jinyeong; Hwang, Wonseok; Yun, Sangdoo; Han, Dongil; Park, Seung-Hyun

doi:10.1007/978-3-031-19815-1_29

Cited by 107 publications

(46 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using visual encoders like ResNet (He et al, 2016), visual features of an input image are also being incorporated into the recent VDU backbones (Xu et al, 2021a;Huang et al, 2022). More recently, with the advances in Vision Transformer (ViT) (Dosovitskiy et al, 2021), training a Transformer encoder-decoder VDU backbone without OCR has also been attempted (Kim et al, 2022;Davis et al, 2022;Lee et al, 2022). Our engine can be used together in various VDU backbones.…”

Section: Vdu Backbonesmentioning

confidence: 99%

“…To construct a rich corpus, in the traditional pipeline, large-scale real-world document images (e.g., IIT-CDIP) and an OCR engine (e.g., CLOVA OCR API 1 ), are used. The quality of the OCR engine significantly affects the downstream processes (Kim et al, 2022;Davis et al, 2022). Hence, there have been difficulties in training and testing the VDU backbone.…”

Section: Visual Corpus Construction For Vdumentioning

confidence: 99%

“…In general, ample visual diversity in the VDU backbone training phase often leads to performance improvement (Jaderberg et al, 2014;Gupta, Vedaldi, and Zisserman, 2016;Yim et al, 2021;Kim et al, 2022).…”

Section: Random Paragraph Fontsmentioning

confidence: 99%

“…Visual Document Understanding (VDU) (Xu et al, 2019;Hong et al, 2022;Kim et al, 2022) has been developed to conduct a wide range of realworld tasks on document images. For example, Document Parsing (Xu et al, 2019;Kim et al, 2022) aims to extract some key texts from a document image (Hwang et al, 2019(Hwang et al, , 2021.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Technical Report on Web-based Visual Corpus Construction for Visual Document Understanding

Kim¹,

Hong²,

Yim³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a dataset generator engine named Web-based Visual Corpus Builder (Webvicob). Webvicob can readily construct a large-scale visual corpus (i.e., images with text annotations) from a raw Wikipedia HTML dump. In this report, we validate that Webvicobgenerated data can cover a wide range of context and knowledge and helps practitioners to build a powerful Visual Document Understanding (VDU) backbone. The proposed engine is publicly available at https:// github.com/clovaai/webvicob.

show abstract

Section: Vdu Backbonesmentioning

confidence: 99%

Section: Visual Corpus Construction For Vdumentioning

confidence: 99%

Section: Random Paragraph Fontsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Technical Report on Web-based Visual Corpus Construction for Visual Document Understanding

Kim¹,

Hong²,

Yim³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The task of visually rich document understanding (VRDU), which involves extracting information from VRDs [2,18], requires models that can handle various types of documents, such as voice, receipts, forms, emails, and advertisements, and various types of information, including rich visuals, large amounts of text, and complex document layouts [27,17,25]. Recently, fine-tuning based on pre-trained visual document understanding models has yielded impressive results in extracting information from VRDs [40,12,21,22,14,20], suggesting that the use of large-scale, unlabeled training documents in pre-training document understanding models can benefit information extraction from VRDs.…”

Section: Introductionmentioning

confidence: 99%

ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction

He¹,

Wang²,

Hu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Large language models (LLMs), such as GPT-3 and ChatGPT, have demonstrated remarkable results in various natural language processing (NLP) tasks with incontext learning, which involves inference based on a few demonstration examples. Despite their successes in NLP tasks, no investigation has been conducted to assess the ability of LLMs to perform document information extraction (DIE) using in-context learning. Applying LLMs to DIE poses two challenges: the modality and task gap. To this end, we propose a simple but effective in-context learning framework called ICL-D3IE, which enables LLMs to perform DIE with different types of demonstration examples. Specifically, we extract the most difficult and distinct segments from hard training documents as hard demonstrations for benefiting all test instances. We design demonstrations describing relationships that enable LLMs to understand positional relationships. We introduce formatting demonstrations for easy answer extraction. Additionally, the framework improves diverse demonstrations by updating them iteratively. Our experiments on three widely used benchmark datasets demonstrate that the ICL-D3IE framework enables GPT-3/ChatGPT to achieve superior performance when compared to previous pretrained methods fine-tuned with full training in both the in-distribution (ID) setting and in the out-of-distribution (OOD) setting.

show abstract

On the Use of Transformers for End-to-End Optical Music Recognition

Ríos-Vila

Iñesta

Calvo-Zaragoza

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

State-of-the-art end-to-end Optical Music Recognition (OMR) has, to date, primarily been carried out using monophonic transcription techniques to handle complex score layouts, such as polyphony, often by resorting to simplifications or specific adaptations. Despite their efficacy, these approaches imply challenges related to scalability and limitations. This paper presents the Sheet Music Transformer (SMT), the first end-to-end OMR model designed to transcribe complex musical scores without relying solely on monophonic strategies. Our model employs a Transformer-based image-to-sequence framework that predicts score transcriptions in a standard digital music encoding format from input images. Our model has been tested on two polyphonic music datasets and has proven capable of handling these intricate music structures effectively. The experimental outcomes not only indicate the competence of the model, but also show that it is better than the state-of-the-art methods, thus contributing to advancements in end-to-end OMR transcription.

show abstract

OCR-Free Document Understanding Transformer

Cited by 107 publications

References 47 publications

Technical Report on Web-based Visual Corpus Construction for Visual Document Understanding

Technical Report on Web-based Visual Corpus Construction for Visual Document Understanding

ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction

On the Use of Transformers for End-to-End Optical Music Recognition

Contact Info

Product

Resources

About