MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics

Zheng, Kunhao; Han, Jesse Michael; Polu, Stanislas

doi:10.48550/arxiv.2109.00110

Cited by 4 publications

(11 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After each stage of fine-tuning, we evaluate the neural theorem prover on miniF2F [54]. The results are shown in Table 3.…”

Section: Resultsmentioning

confidence: 99%

“…MiniF2F [54] is a recently introduced benchmark containing 488 mathematical competition statements manually formalized by humans in three different formal languages. Its goal is to compare and benchmark methods across different theorem provers for machine learning research.…”

Section: Mathematical Competition Datasetsmentioning

confidence: 99%

“…We use autoformalized statements as targets for proof search with a neural theorem prover for Isabelle/HOL. After fine-tuning our neural theorem prover on the proofs it found, its success rate on the MiniF2F benchmark [54] increases significantly, achieving a new state-of-the-art result of 35.2% theorems proven.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Autoformalization with Large Language Models

Wu¹,

Jiang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Autoformalization is the process of automatically translating from natural language mathematics to formal specifications and proofs. A successful autoformalization system could advance the fields of formal verification, program synthesis, and artificial intelligence. While the long-term goal of autoformalization seemed elusive for a long time, we show large language models provide new prospects towards this goal. We make the surprising observation that LLMs can correctly translate a significant portion (25.3%) of mathematical competition problems perfectly to formal specifications in Isabelle/HOL. We demonstrate the usefulness of this process by improving a previously introduced neural theorem prover via training on these autoformalized theorems. Our methodology results in a new state-of-the-art result on the MiniF2F theorem proving benchmark, improving the proof rate from 29.6% to 35.2%.Recent advances in large language models [7,9] showed promising capabilities of understanding formal languages [8,32]. However, the existing successes are limited to formal languages where there exists a large body of corpus on the web (e.g., Python language). Formal mathematics data is very scarce. For example, one of the largest formal mathematics libraries, the Archive of Formal Proofs, is only 180MB in size, that is less than 0.18% of the training data for the large language model Codex [8]. Moreover, unlike in the case of commonly used programming languages, where natural language docstrings are broadly available, there is almost zero aligned data between natural language and formal mathematics. Therefore, it is unclear the recent successes can directly contribute to the development of autoformalization.In this work, we explore the prospects of autoformalization with large language models. To our surprise, we find that large language models already have a decent capability of formalizing natural † Correspondence to Yuhuai Wu

show abstract

“…After each stage of fine-tuning, we evaluate the neural theorem prover on miniF2F [54]. The results are shown in Table 3.…”

Section: Resultsmentioning

confidence: 99%

Section: Mathematical Competition Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Autoformalization with Large Language Models

Wu¹,

Jiang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In Polu et al [18], the model is fine-tuned on theorems from the training set and expert iteration is done on theorems from different sources: train theorems, synthetic statements, and an extra curriculum of statements without proofs (miniF2F-curriculum). The produced model is then evaluated on unseen statements, namely the validation and test splits of the miniF2F dataset [8].…”

Section: Evaluation Settings and Protocolmentioning

confidence: 99%

“…• State-of-the-art performance on all analyzed environments. In particular, our model manages to prove over 82.6% of proofs in a held-out set of theorems from set.mm in Metamath, as well as 58.6% on miniF2F-valid [8] in Lean.…”

Section: Introductionmentioning

confidence: 99%

HyperTree Proof Search for Neural Theorem Proving

Lample¹,

Lachaux²,

Lavril³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose an online training procedure for a transformer-based automated theorem prover. Our approach leverages a new search algorithm, HyperTree Proof Search (HTPS), inspired by the recent success of AlphaZero. Our model learns from previous proof searches through online training, allowing it to generalize to domains far from the training distribution. We report detailed ablations of our pipeline's main components by studying performance on three environments of increasing complexity. In particular, we show that with HTPS alone, a model trained on annotated proofs manages to prove 65.4% of a held-out set of Metamath theorems, significantly outperforming the previous state of the art of 56.5% by GPT-f. Online training on these unproved theorems increases accuracy to 82.6%. With a similar computational budget, we improve the state of the art on the Lean-based miniF2F-curriculum dataset from 31% to 42% proving accuracy.

show abstract

Solving olympiad geometry without human demonstrations

Trinh,

Wu,

et al. 2024

Nature

View full text Add to dashboard Cite

Proving mathematical theorems at the olympiad level represents a notable milestone in human-level automated reasoning1–4, owing to their reputed difficulty among the world’s best talents in pre-university mathematics. Current machine-learning approaches, however, are not applicable to most mathematical domains owing to the high cost of translating human proofs into machine-verifiable format. The problem is even worse for geometry because of its unique translation challenges1,5, resulting in severe scarcity of training data. We propose AlphaGeometry, a theorem prover for Euclidean plane geometry that sidesteps the need for human demonstrations by synthesizing millions of theorems and proofs across different levels of complexity. AlphaGeometry is a neuro-symbolic system that uses a neural language model, trained from scratch on our large-scale synthetic data, to guide a symbolic deduction engine through infinite branching points in challenging problems. On a test set of 30 latest olympiad-level problems, AlphaGeometry solves 25, outperforming the previous best method that only solves ten problems and approaching the performance of an average International Mathematical Olympiad (IMO) gold medallist. Notably, AlphaGeometry produces human-readable proofs, solves all geometry problems in the IMO 2000 and 2015 under human expert evaluation and discovers a generalized version of a translated IMO theorem in 2004.

show abstract

MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics

Cited by 4 publications

References 18 publications

Autoformalization with Large Language Models

Autoformalization with Large Language Models

HyperTree Proof Search for Neural Theorem Proving

Solving olympiad geometry without human demonstrations

Contact Info

Product

Resources

About