From Dataset Recycling to Multi-Property Extraction and Beyond

Abstract

This paper investigates various Transformer architectures on the WikiReading Information Extraction and Machine Reading Comprehension dataset. The proposed dual-source model outperforms the current state-of-theart by a large margin.Next, we introduce WikiReading Recycled-a newly developed public dataset, and the task of multipleproperty extraction. It uses the same data as WikiReading but does not inherit its predecessor's identified disadvantages. In addition, we provide a human-annotated test set with diagnostic subsets for a detailed analysis of model performance.

Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer

“…Sequence labeling models can be trained in all cases where the token-level annotation is available or can be easily obtained. Limitations of this approach are strikingly visible on tasks framed in either key information extraction or property extraction paradigms [20,9]. Here, no annotated spans are available, and only property-value pairs are assigned to the document.…”

“…Finally, the property extraction paradigm does not assume the requested value appeared in the article in any form since it is sufficient for it to be inferable from the content, as in document classification or non-extractive question answering [9].…”

“…The QA program of unifying NLP frames all the problems as triplets of question, context and answer [30,40,27] or item, property name and answer [17]. Although this does not necessarily lead to the use of encoder-decoder models, several successful solutions relied on variants of Transformer architecture [54,35,9,45]. The T5 is a prominent example of large-scale Transformers achieving state-of-the-art results on varied NLP benchmarks [45].…”

Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts

“…The WikiReading dataset [8] (and its variant WikiReading Recycled [6]) is a large-scale natural language understanding task. Here, the main goal is to predict textual values from the structured knowledge base, Wikidata, by reading the text of the corresponding Wikipedia articles.…”

“…This disparity is still large and makes a robust evaluation difficult. Recently, researchers have started to fill the gap by creating datasets in the KIE domain such as scanned receipts: SROIE 5 [18], form understanding [11], NIST Structured Forms Reference Set of Binary Images (SFRS ) 6 or Visual Question Answering dataset DocVQA [15].…”