Are NLP Models really able to Solve Simple Math Word Problems?

Abstract

The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered "solved" with the bulk of research attention moving to more complex MWPs. In this paper, we restrict our attention to English MWPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. Similarly, models that treat MWPs as bag-ofwords can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs. PROBLEM: Jack had 8 pens and Mary had 5 pens. Jack gave 3 pens to Mary. How many pens does Jack have now? 8 -3 = 5 QUESTION SENSITIVITY VARIATION: Jack had 8 pens and Mary had 5 pens. Jack gave 3 pens to Mary. How many pens does Mary have now? 5 + 3 = 8 REASONING ABILITY VARIATION: Jack had 8 pens and Mary had 5 pens. Mary gave 3 pens to Jack. How many pens does Jack have now? 8 + 3 = 11 STRUCTURAL INVARIANCE VARIATION: Jack gave 3 pens to Mary. If Jack had 8 pens and Mary had 5 pens initially, how many pens does Jack have now? 8 -3 = 5

Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problems

“…However, it is evident that neural networks are black boxes, and hard to interpret their functioning and explain their decision . Attempts to interpret its functioning reveal how volatile its reasoning ability is, and they lack generalizability (Patel et al, 2021).…”

“…Similarly, Dolphin1878 comprises of curated problems from both algebra.com and answers.yahoo.com. Derived datasets have been derived by processing existing datasets to address concerns related to training corpus size (Roy and Roth, 2017), equation annotation (Amini et al, 2019), lexical diversity (Miao et al, 2021), problem types (Roy and Roth, 2017) or adversarial reasoning ability (Patel et al, 2021). Synthetic datasets could be very large, but they lack generalization.…”

“…Lexicon Diversity Score measures the lexical diversity of questions provided in the dataset, and datasets with better lexicon diversity score would be more reliable to test a model's performance (Miao et al, 2021). Also, adversarial approaches further extend such datasets with adversarial variants to test the mathematical reasoning ability (Patel et al, 2021). (Wang et al, 2017) proposed the first encoderdecoder architecture to generate equations or expression trees for MWPs.…”

“…Large datasets have empowered neural models to learn complex mathematical concepts like integration, differentiation (Lample and Charton, 2019) (Saxton et al, 2019), etc, but such models often overfit on smaller datasets (Miao et al, 2021) (Patel et al, 2021. Data augmentation is a popular preprocessing technique to increase the size of training data.…”

“…Different traversal orders of expression trees and other preprocessing techniques like lemmatization, POS tagging, sentence reordering, stop word removal, etc., and their impact on the performance of models have been studied by (Griffith and Kalita, 2021). Data processing and augmentation could also help to generate adversarial datasets, which could test the reasoning ability of models (Patel et al, 2021)(Miao et al, 2021. Weak supervision is another popular technique that generates large datasets with noisy labels.…”