Deep Generation of Coq Lemma Names Using Elaborated Terms

Abstract

Coding conventions for naming, spacing, and other essentially stylistic properties are necessary for developers to effectively understand, review, and modify source code in large software projects. Consistent conventions in verification projects based on proof assistants, such as Coq, increase in importance as projects grow in size and scope. While conventions can be documented and enforced manually at high cost, emerging approaches automatically learn and suggest idiomatic names in Java-like languages by applying statistical language models on large code corpora. However, due to its powerful language extension facilities and fusion of type checking and computation, Coq is a challenging target for automated learning techniques. We present novel generation models for learning and suggesting lemma names for Coq projects. Our models, based on multi-input neural networks, are the first to leverage syntactic and semantic information from Coq's lexer (tokens in lemma statements), parser (syntax trees), and kernel (elaborated terms) for naming; the key insight is that learning from elaborated terms can substantially boost model performance. We implemented our models in a toolchain, dubbed Roosterize, and applied it on a large corpus of code derived from the Mathematical Components family of projects, known for its stringent coding conventions. Our results show that Roosterize substantially outperforms baselines for suggesting lemma names, highlighting the importance of using multi-input models and elaborated terms.

Roosterize: Suggesting Lemma Names for Coq Verification Projects Using Deep Learning

“…We evaluated an earlier version of ROOSTERIZE using a corpus derived from the MathComp family of Coq projects, finding that the toolchain significantly outperforms strong baselines on automatic metrics [7]. Moreover, we found encouraging results in a qualitative case study where the maintainer of a medium-sized Coq project manually evaluated over 150 name suggestions generated by ROOSTERIZE.…”

“…Users should first obtain a model, e.g., by downloading a pre-trained model. The following command downloads the model we pre-trained on our MathComp corpus [7]:…”

“…ROOSTERIZE learns naming conventions by leveraging neural networks trained on existing Coq code. The deep learning and suggestion processes use multiple representations of lemma statements, including syntax trees and Coq kernel trees (also called elaborated terms) [7]. In essence, ROOSTERIZE consists of (1) a set of components written in OCaml that interact with Coq or directly process information extracted from Coq, and (2) a set of components written in Python that perform name learning and generation.…”

Learning to Format Coq Code Using Language Models

“…Despite the high accuracy achieved by our preliminary implementation even when using the baseline n-gram model, we believe our spacing prediction (based only on raw token streams) needs significant tuning for practical use. For example, newlines before Qed sentences often get mispredicted, and unlike for name suggestions [3], it is usually inconvenient to inspect more than the top-1 suggestion for spacing. Moreover, for MathComp, we were able to construct, with help from maintainers, a sufficiently large corpus with strict adherence to conventions; for other projects, it may be more challenging, e.g., due to project size or lack of consensus on conventions.…”

“…As a first step, we here outline initial models to learn and suggest space formatting in Coq files, with a preliminary implementation for Coq 8.10, and evaluated using on a corpus based on MathComp 1.9.0 which comprises 164k lines of Coq code from four core projects [3].…”