Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension

Abstract

Reading comprehension models have been successfully applied to extractive text answers, but it is unclear how best to generalize these models to abstractive numerical answers. We enable a BERT-based reading comprehension model to perform lightweight numerical reasoning. We augment the model with a predefined set of executable 'programs' which encompass simple arithmetic as well as extraction. Rather than having to learn to manipulate numbers directly, the model can pick a program and execute it. On the recent Discrete Reasoning Over Passages (DROP) dataset, designed to challenge reading comprehension models, we show a 33% absolute improvement by adding shallow programs. The model can learn to predict new operations when appropriate in a math word problem setting (Roy and Roth, 2015) with very few training examples.

A Simple and Effective Model for Answering Multi-span Questions

“…Hu et al (2019) proposed to predict the number of output spans for each question, and used a non-differentiable inference procedure to find them in the text, leading to a complex training procedure. Andor et al (2019) proposed a Merge operation that merges spans, but is constrained to at most 2 spans. Chen et al (2020) proposed a non-differentiable symbolic approach which outputs programs that compose single-span extractions.…”

Question Directed Graph Attention Network for Numerical Reasoning over Text

“…The overall experimental results are reported in Table 2, where the performance of baseline methods is obtained from previous work (Dua et al, 2019;Seo et al, 2017;Ran et al, 2019;Andor et al, 2019) and the public leaderboard. 6 The first three methods in Table 2 are based on either semantic parsing or information extraction, and perform poorly on the numerical MRC task.…”

“…To improve this method, Ran et al (2019) proposed NumNet, which constructs a number comparison graph that encodes the relative magnitude information between numbers on directed edges. Although NumNet achieves superior performance than other numerically-aware models (Hu et al, 2019a;Andor et al, 2019;Geva et al, 2020;, we argue that NumNet is insufficient for sophisticated numerical reasoning, since it lacks two critical ingredients for numerical reasoning:…”

“…provides a semantic parser that points to locations in the text that can be used in further numerical operations. BERT-Calculator (Andor et al, 2019) defines a set of executable programs and learns to choose one to derive numerical answers. NumNet (Ran et al, 2019) uses a numerically-aware graph neural network to encode numbers, which made further progress on the DROP dataset.…”

“…(1) NAQANet (Dua et al, 2019), a model that adapts the output layer of QANet to numeric reasoning; (2) ALBERT-Calculator (Andor et al, 2019), a model based on ALBERT-xxlarge (Lan et al, 2020) that picks one of executable programs from a predefined set to derive numerical answers. ( 3) NumNet, a model that embeds numerical properties into the distributed representation by using a GNN on the number graph;…”