Autoformalization with Large Language Models

Abstract

Autoformalization is the process of automatically translating from natural language mathematics to formal specifications and proofs. A successful autoformalization system could advance the fields of formal verification, program synthesis, and artificial intelligence. While the long-term goal of autoformalization seemed elusive for a long time, we show large language models provide new prospects towards this goal. We make the surprising observation that LLMs can correctly translate a significant portion (25.3%) of mathematical competition problems perfectly to formal specifications in Isabelle/HOL. We demonstrate the usefulness of this process by improving a previously introduced neural theorem prover via training on these autoformalized theorems. Our methodology results in a new state-of-the-art result on the MiniF2F theorem proving benchmark, improving the proof rate from 29.6% to 35.2%.Recent advances in large language models [7,9] showed promising capabilities of understanding formal languages [8,32]. However, the existing successes are limited to formal languages where there exists a large body of corpus on the web (e.g., Python language). Formal mathematics data is very scarce. For example, one of the largest formal mathematics libraries, the Archive of Formal Proofs, is only 180MB in size, that is less than 0.18% of the training data for the large language model Codex [8]. Moreover, unlike in the case of commonly used programming languages, where natural language docstrings are broadly available, there is almost zero aligned data between natural language and formal mathematics. Therefore, it is unclear the recent successes can directly contribute to the development of autoformalization.In this work, we explore the prospects of autoformalization with large language models. To our surprise, we find that large language models already have a decent capability of formalizing natural † Correspondence to Yuhuai Wu

Improving the Diproche CNL through Autoformalization via Large Language Models

“…); further sub-types include declarations combined with assumptions, such as "Let k be an integer such that n = 2k" (which are existentially loaded and are in need of verification) and justified claims ("Since x = 3(a + b), x is a multiple of 3"). The sentences 12 written in this CNL are converted into an internal list format whose crucial ingredients are a list of the of variables occuring in the sentence, its type (assumption, declaration, claim, annotation,...) and its actual content (which can be empty, as in the case of annotations). Thus, the sentence "Therefore, x is even" would be translated as…”

“…At first, the experiences reported in Avigad et al [1] with using large language models for autoformalization appear to be discouraging for this plan: Only about 11 percent of the natural language inputs were formalized correctly ( [1], p. 3). Much better results were reported in [12], where more than 25 percent of the natural language inputs (which were problems for math competitions) were translated correctly into Isabelle (ibid., p. 1). Still, for reliably checking even a simple natural language argument consisting of typically more than 10 sentences with sufficient reliability to be of didactical use to beginner's students, anything considerably below a hundred percent is not good enough.…”

nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

“…LLMs are Transformers [43], which is the state of the art neural architecture for natural language proccessing. Additionally, Transformers have shown remarkable performance when being applied to classical problems in verification (e.g., [19,41,26,9]), reasoning (e.g., [28,51]), as well as the auto-formalization [36] of mathematics and formal specifications (e.g., [50,20,22]).…”

Formal Specifications from Natural Language

“…The term autoformalization (Wang et al, 2018;Szegedy, 2020) has been coined for tasks of translating between natural language and formal specifications or proofs. Closest to our work is a very recent, independently developed, effort in translating between natural language and formal proofs using very large language models (Wu et al, 2022).…”