Proof Artifact Co-training for Theorem Proving with Language Models

Han, Jesse Michael; Rute, Jason; Wu, Yuhuai; Ayers, Edward W.; Polu, Stanislas

doi:10.48550/arxiv.2102.06203

Cited by 7 publications

(20 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, in order to study baseline performances associated with existing systems, we report pass rates achieved by GPT-f [12] applied to Metamath, GPT-f/PACT [12,7] applied to Lean as well as a baseline prover implemented in Lean denoted as the tidy baseline. Pass rates are reported as Pass@N where N is the number of proof search attempts per statement.…”

Section: Methodsmentioning

confidence: 99%

“…They also proposed a methodology to train a value function to further guide proof search, achieving a 56.22% pass rate on the held-out test set. Large language models were applied to Lean by Han et al [7]. They created an environment around the Lean prover targeted to machine learning and propose a dataset extracted from low level proof artifacts that is shown to boost performance when used as a self-supervised co-training objective.…”

Section: Neural Theorem Provingmentioning

confidence: 99%

“…We use the generic best-first search algorithm presented in PACT [7]. The algorithm works as follows: Given a list of tactics L with priority, we maintain a priority queue Q of tactic states whose priority is given by the priority of the last applied tactic in L that led to it.…”

Section: Tidy Baselinementioning

confidence: 99%

“…Neural theorem proving is a rapidly developing area which aims to apply techniques from deep learning to interactive theorem proving. To date, most contributions in this area have focused on individual theorem proving systems, each with a separately-implemented mathematics library and with results reported on a dataset-specific test split; examples include the HOList [2], CoqGym [24] and LeanStep [7] theorem proving environments and benchmarks. However, benchmarks from this paradigm are not ideal for measuring the mathematical reasoning ability of neural theorem provers for several reasons.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics

Zheng

Han

Polu

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We present miniF2F, a dataset of formal Olympiad-level mathematics problems statements intended to provide a unified cross-system benchmark for neural theorem proving. The miniF2F benchmark currently targets Metamath, Lean, and Isabelle and consists of 488 problem statements drawn from the AIME, AMC, and the International Mathematical Olympiad (IMO), as well as material from high-school and undergraduate mathematics courses. We report baseline results using GPT-f [12], a neural theorem prover based on and provide an analysis of its performance. We intend for miniF2F to be a community-driven effort and hope that our benchmark will help spur advances in neural theorem proving.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Neural Theorem Provingmentioning

confidence: 99%

Section: Tidy Baselinementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics

Zheng

Han

Polu

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For automated theorem proving and mathematical reasoning, various datasets and accompanying approaches have been proposed (Kaliszyk et al, 2017;Bansal et al, 2019) including more recent work with equational logic (Piepenbrock et al, 2021) and language models (Rabe et al, 2020;Han et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics

Jordan¹,

Zhou²,

Freitas³

2022

Preprint

View full text Add to dashboard Cite

In order for language models to aid physics research, they must first encode representations of mathematical and natural language discourse which lead to coherent explanations, with correct ordering and relevance of statements. We present a collection of datasets developed to evaluate the performance of language models in this regard, which measure capabilities with respect to sentence ordering, position, section prediction, and discourse coherence. Analysis of the data reveals equations and sub-disciplines which are most common in physics discourse, as well as the sentence-level frequency of equations and expressions. We present baselines which demonstrate how contemporary language models are challenged by coherence related tasks in physics, even when trained on mathematical natural language objectives.

show abstract

Solving olympiad geometry without human demonstrations

Trinh,

Wu,

et al. 2024

Nature

Self Cite

View full text Add to dashboard Cite

Proving mathematical theorems at the olympiad level represents a notable milestone in human-level automated reasoning1–4, owing to their reputed difficulty among the world’s best talents in pre-university mathematics. Current machine-learning approaches, however, are not applicable to most mathematical domains owing to the high cost of translating human proofs into machine-verifiable format. The problem is even worse for geometry because of its unique translation challenges1,5, resulting in severe scarcity of training data. We propose AlphaGeometry, a theorem prover for Euclidean plane geometry that sidesteps the need for human demonstrations by synthesizing millions of theorems and proofs across different levels of complexity. AlphaGeometry is a neuro-symbolic system that uses a neural language model, trained from scratch on our large-scale synthetic data, to guide a symbolic deduction engine through infinite branching points in challenging problems. On a test set of 30 latest olympiad-level problems, AlphaGeometry solves 25, outperforming the previous best method that only solves ten problems and approaching the performance of an average International Mathematical Olympiad (IMO) gold medallist. Notably, AlphaGeometry produces human-readable proofs, solves all geometry problems in the IMO 2000 and 2015 under human expert evaluation and discovers a generalized version of a translated IMO theorem in 2004.

show abstract

Proof Artifact Co-training for Theorem Proving with Language Models

Cited by 7 publications

References 27 publications

MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics

MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics

PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics

Solving olympiad geometry without human demonstrations

Contact Info

Product

Resources

About