Analysing Mathematical Reasoning Abilities of Neural Models

Saxton, David; Grefenstette, Edward; Hill, Felix; Kohli, Pushmeet

doi:10.48550/arxiv.1904.01557

Cited by 35 publications

(54 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Saxton et al (2019) they study generalization on procedurally generated math problems such as arithmetic and differentiation, but for the most part these tasks are more involved than the simple binary op problems we have studied and as such do not lend themselves to observing the kinds of phenomena we describe in this paper, since they would require an extremely large number of samples to master.…”

Section: A3 Related Workmentioning

confidence: 99%

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Power¹,

Burda²,

Edwards³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of learning can be studied in great detail. In some situations we show that neural networks learn through a process of "grokking" a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting. We also study generalization as a function of dataset size and find that smaller datasets require increasing amounts of optimization for generalization. We argue that these datasets provide a fertile ground for studying a poorly understood aspect of deep learning: generalization of overparametrized neural networks beyond memorization of the finite training dataset.

show abstract

Section: A3 Related Workmentioning

confidence: 99%

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Power¹,

Burda²,

Edwards³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Transformers have also been applied to formal systems, in theorem proving (Polu & Sutskever, 2020) and temporal logic (Hahn et al, 2021). The use of sequence to sequence models for arithmetic and the exact solution of mathematical problem has been studied by Saxton et al (2019).…”

Section: Related Workmentioning

confidence: 99%

Linear algebra with transformers

Charton¹

2021

Preprint

View full text Add to dashboard Cite

Most applications of transformers to mathematics, from integration to theorem proving, focus on symbolic computation. In this paper, we show that transformers can be trained to perform numerical calculations with high accuracy. We consider problems of linear algebra: matrix transposition, addition, multiplication, eigenvalues and vectors, singular value decomposition, and inversion. Training small transformers (up to six layers) over datasets of random matrices, we achieve high accuracies (over 90%) on all problems. We also show that trained models can generalize out of their training distribution, and that out-of-domain accuracy can be greatly improved by working from more diverse datasets (in particular, by training from matrices with non-independent and identically distributed coefficients). Finally, we show that few-shot learning can be leveraged to re-train models to solve larger problems.

show abstract

“…Mathematical Reasoning In the area of mathematical reasoning, several projects have probed the limits of transformers to solve pure math problems (Saxton et al, 2019;Lample and Charton, 2019;Hendrycks et al, 2021). FPs differ from these problems in two important ways.…”

Section: Related Workmentioning

confidence: 99%

How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI

Kalyan¹,

Kumar²,

Chandrasekaran³

et al. 2021

Preprint

View full text Add to dashboard Cite

Many real-world problems require the combined application of multiple reasoning abilities-employing suitable abstractions, commonsense knowledge, and creative synthesis of problem-solving strategies. To help advance AI systems towards such capabilities, we propose a new reasoning challenge, namely Fermi Problems (FPs), which are questions whose answers can only be approximately estimated because their precise computation is either impractical or impossible. For example, "How much would the sea level rise if all ice in the world melted?" FPs are commonly used in quizzes and interviews to bring out and evaluate the creative reasoning abilities of humans. To do the same for AI systems, we present two datasets: 1) A collection of 1k real-world FPs sourced from quizzes and olympiads; and 2) a bank of 10k synthetic FPs of intermediate complexity to serve as a sandbox for the harder real-world challenge. In addition to question-answer pairs, the datasets contain detailed solutions in the form of an executable program and supporting facts, helping in supervision and evaluation of intermediate steps. We demonstrate that even extensively fine-tuned large-scale language models perform poorly on these datasets, on average making estimates that are off by two orders of magnitude. Our contribution is thus the crystallization of several unsolved AI problems into a single, new challenge that we hope will spur further advances in building systems that can reason.

show abstract

Analysing Mathematical Reasoning Abilities of Neural Models

Cited by 35 publications

References 18 publications

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Linear algebra with transformers

How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI

Contact Info

Product

Resources

About