DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

Dua, Dheeru; Wang, Yizhong; Dasigi, Pradeep; Stanovsky, Gabriel; Singh, Sameer; Gardner, Matt

doi:10.48550/arxiv.1903.00161

Cited by 37 publications

(53 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Progress towards deeper numerical representations has been limited but promising. For example, previous work, represented by the DROP (Dua et al 2019), presents a variety of numerical reasoning problems. Different from the existing works that pay attention to explore the capability of pretrained language models for general common-sense reasoning (Zhang et al 2020), and math word problem-solving (Wu et al 2021), we focus on improving the numeral understanding ability of language models for financial forecasting, motivated by the fact that financial documents often contain massive amounts of numbers.…”

Section: Representing Numbers In Language Modelsmentioning

confidence: 99%

NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task Financial Forecasting

Yang¹,

Li²,

Dong³

et al. 2022

Preprint

View full text Add to dashboard Cite

Financial forecasting has been an important and active area of machine learning research because of the challenges it presents and the potential rewards that even minor improvements in prediction accuracy or forecasting may entail. Traditionally, financial forecasting has heavily relied on quantitative indicators and metrics derived from structured financial statements. Earnings conference call data, including text and audio, is an important source of unstructured data that has been used for various prediction tasks using deep earning and related approaches. However, current deep learning-based methods are limited in the way that they deal with numeric data; numbers are typically treated as plaintext tokens without taking advantage of their underlying numeric structure. This paper describes a numericoriented hierarchical transformer model (NumHTML) to predict stock returns, and financial risk using multimodal aligned earnings calls data by taking advantage of the different categories of numbers (monetary, temporal, percentages etc.) and their magnitude. We present the results of a comprehensive evaluation of NumHTML against several state-of-the-art baselines using a real-world publicly available dataset. The results indicate that NumHTML significantly outperforms the current state-of-the-art across a variety of evaluation metrics and that it has the potential to offer significant financial gains in a practical trading context.

show abstract

Section: Representing Numbers In Language Modelsmentioning

confidence: 99%

NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task Financial Forecasting

Yang¹,

Li²,

Dong³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Similar datasets follow this trend of using free-form questions and adopt reading documents from a variety of sources, such as news articles [37,17] and dialogues [18,28,4]. In addition to these datasets where the answers can be directly extracted from the document, another popular type of datasets, i.e., abstractive datasets, ask the reader to generate an answer that may not be found in the given context [23,12]. Abstractive datasets further require the reader to perform non-trivial reasoning over the facts in the document as well as common sense knowledge to produce answers.…”

Section: Related Workmentioning

confidence: 99%

Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension

Xu¹,

Yi-chen²,

Yi³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present Native Chinese Reader (NCR), a new machine reading comprehension (MRC) dataset with particularly long articles in both modern and classical Chinese. NCR is collected from the exam questions for the Chinese course in China's high schools, which are designed to evaluate the language proficiency of native Chinese youth. Existing Chinese MRC datasets are either domain-specific or focusing on short contexts of a few hundreds of characters in modern Chinese only. By contrast, NCR contains 8390 documents with an average length of 1024 characters covering a wide range of Chinese writing styles, including modern articles, classical literature and classical poetry. A total of 20477 questions on these documents also require strong reasoning abilities and common sense to figure out the correct answers. We implemented multiple baseline models using popular Chinese pretrained models and additionally launched an online competition using our dataset to examine the limit of current methods. The best model achieves 59% test accuracy while human evaluation shows an average accuracy of 79%, which indicates a significant performance gap between current MRC models and native Chinese speakers. We release the dataset at https://sites.google.com/ view/native-chinese-reader/.

show abstract

“…d) Numerical Reasoning: Numerical reasoning involves performing discrete arithmetic reasoning over quantities to solve mathematical word problems [42,43,44,45], which is a fundamental and challenging task. Previous work translates the textual math word problem into an expression or an expression tree and utilizes arithmetic knowledge to solve it [43,46,47].…”

Section: A Taxonomy Of Complex Reasoningmentioning

confidence: 99%

“…Logical reasoning extends complex reasoning ability with logical deduction over propositions while analytical reasoning simulates the human analytical thinking and problemsolving capacity. Then several widely studied complex reasoning tasks should be integrated, such as reasoning for commonsense knowledge [27,29,30], multi-hop relationships [35,36] and numerical calculation [44,45] described in § II-A. Some other rarely explored complex reasoning abilities also need to be considered.…”

Section: B Challenges and Future Directionsmentioning

confidence: 99%

From LSAT: The Progress and Challenges of Complex Reasoning

Wang¹,

Liu²,

Zhong³

et al. 2021

Preprint

View full text Add to dashboard Cite

Complex reasoning aims to draw a correct inference based on complex rules. As a hallmark of human intelligence, it involves a degree of explicit reading comprehension, interpretation of logical knowledge and complex rule application. In this paper, we take a step forward in complex reasoning by systematically studying the three challenging and domain-general tasks of the Law School Admission Test (LSAT), including analytical reasoning, logical reasoning and reading comprehension. We propose a hybrid reasoning system to integrate these three tasks and achieve impressive overall performance on the LSAT tests. The experimental results demonstrate that our system endows itself a certain complex reasoning ability, especially the fundamental reading comprehension and challenging logical reasoning capacities. Further analysis also shows the effectiveness of combining the pre-trained models with the task-specific reasoning module, and integrating symbolic knowledge into discrete interpretable reasoning steps in complex reasoning. We further shed a light on the potential future directions, like unsupervised symbolic knowledge extraction, model interpretability, few-shot learning and comprehensive benchmark for complex reasoning.

show abstract

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

Cited by 37 publications

References 0 publications

NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task Financial Forecasting

NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task Financial Forecasting

Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension

From LSAT: The Progress and Challenges of Complex Reasoning

Contact Info

Product

Resources

About