Are NLP Models really able to Solve Simple Math Word Problems?

Patel, Arkil; Bhattamishra, Satwik; Goyal, Navin

doi:10.18653/v1/2021.naacl-main.168

Cited by 108 publications

(129 citation statements)

References 22 publications

(28 reference statements)

Supporting

Mentioning

129

Contrasting

Order By: Relevance

“…The tokenization scheme could be the cause for limited extrapolation, since language models get better at arithmetic when numbers are tokenized at the digit/character level (Nogueira et al, 2021;Wallace et al, 2019). For arithmetic word problems, state of the art solvers rely on predicting an equation, which is then filled in with specific numeric values from the question (Patel et al, 2021), altogether bypassing the need for encoding numbers into embeddings.…”

Section: Resultsmentioning

confidence: 99%

Representing Numbers in NLP: a Survey and a Vision

Thawani¹,

Pujara²,

Szekely³

et al. 2021

Preprint

View full text Add to dashboard Cite

NLP systems rarely give special consideration to numbers found in text. This starkly contrasts with the consensus in neuroscience that, in the brain, numbers are represented differently from words. We arrange recent NLP work on numeracy into a comprehensive taxonomy of tasks and methods. We break down the subjective notion of numeracy into 7 subtasks, arranged along two dimensions: granularity (exact vs approximate) and units (abstract vs grounded). We analyze the myriad representational choices made by 18 previously published number encoders and decoders. We synthesize best practices for representing numbers in text and articulate a vision for holistic numeracy in NLP, comprised of design trade-offs and a unified evaluation.

show abstract

Section: Resultsmentioning

confidence: 99%

Representing Numbers in NLP: a Survey and a Vision

Thawani¹,

Pujara²,

Szekely³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Datasets We conduct experiments on four datasets across two different languages: MAWPS (Koncel-Kedziorski et al, 2016), Math23k (Wang et al, 2017), MathQA (Amini et al, 2019), and SVAMP (Patel et al, 2021). The dataset statistics can be found in Table 2…”

Section: Methodsmentioning

confidence: 99%

“…S2T/G2T GTS (Xie and Sun, 2019) 82.6 Graph2Tree 85.6 Roberta-GTS (Patel et al, 2021) 88.5 Roberta-Graph2Tree (Patel et al, 2021) 88 adapt the dataset to filter out some questions that are unsolvable. We consider the operations "addition", "subtraction", "multiplication", and "division" for MAWPS and SVAMP, and an extra "exponentiation" for MathQA and Math23k.…”

Section: S2smentioning

confidence: 99%

See 1 more Smart Citation

Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction

Jie¹,

Jierui²,

Lü³

2022

Preprint

View full text Add to dashboard Cite

Solving math word problems requires deductive reasoning over the quantities in the text. Various recent research efforts mostly relied on sequence-to-sequence or sequence-to-tree models to generate mathematical expressions without explicitly performing relational reasoning between quantities in the given context. While empirically effective, such approaches typically do not provide explanations for the generated expressions. In this work, we view the task as a complex relation extraction problem, proposing a novel approach that presents explainable deductive reasoning steps to iteratively construct target expressions, where each step involves a primitive operation over two quantities defining their relation. Through extensive experiments on four benchmark datasets, we show that the proposed model significantly outperforms existing strong baselines. We further demonstrate that the deductive procedure not only presents more explainable steps but also enables us to make more accurate predictions on questions that require more complex reasoning.

show abstract

“…However, while using deep learning to solve MWPs, existing methods (Xie and Sun, 2019;Zhang et al, 2020) get stuck in memorizing procedures. Patel et al (2021) provide evidence that these methods rely on shallow heuristics to generate equations. We look at this issue and think it is because they focus on text understanding or equation generation for one problem.…”

Section: Introductionmentioning

confidence: 99%

Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems

Li¹,

Zhang²,

Yan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Math Word Problem (MWP) solving needs to discover the quantitative relationships over natural language narratives. Recent work shows that existing models memorize procedures from context and rely on shallow heuristics to solve MWPs. In this paper, we look at this issue and argue that the cause is a lack of overall understanding of MWP patterns. We first investigate how a neural network understands patterns only from semantics, and observe that, if the prototype equations like n 1 + n 2 are the same, most problems get closer representations and those representations apart from them or close to other prototypes tend to produce wrong solutions. Inspired by it, we propose a contrastive learning approach, where the neural network perceives the divergence of patterns. We collect contrastive examples by converting the prototype equation into a tree and seeking similar tree structures. The solving model is trained with an auxiliary objective on the collected examples, resulting in the representations of problems with similar prototypes being pulled closer. We conduct experiments 1 on the Chinese dataset Math23k and the English dataset MathQA. Our method greatly improves the performance in monolingual and multilingual settings.

show abstract

Are NLP Models really able to Solve Simple Math Word Problems?

Cited by 108 publications

References 22 publications

Representing Numbers in NLP: a Survey and a Vision

Representing Numbers in NLP: a Survey and a Vision

Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction

Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems

Contact Info

Product

Resources

About