CodeT: Code Generation with Generated Tests

Abstract

Given a programming problem, pre-trained language models such as Codex have demonstrated the ability to generate multiple different code solutions via sampling. However, selecting a correct or best solution from those samples still remains a challenge. While an easy way to verify the correctness of a code solution is through executing test cases, producing high-quality test cases is prohibitively expensive. In this paper, we explore the use of pre-trained language models to automatically generate test cases, calling our method CODET: CODE generation with generated Tests. CODET executes the code solutions using the generated test cases, and then chooses the best solution based on a dual execution agreement with both the generated test cases and other generated solutions. We evaluate CODET on five different pre-trained models with both HumanEval and MBPP benchmarks. Extensive experimental results demonstrate CODET can achieve significant, consistent, and surprising improvements over previous methods. For example, CODET improves the pass@1 on HumanEval to 65.8%, an increase of absolute 18.8% on the code-davinci-002 model, and an absolute 20+% improvement over previous state-of-the-art results. * The first three authors contributed equally. 1 https://github.com/features/copilot 2 Results on the HumanEval benchmark with code-cushman-001. More results can be found in Section 4.1.

Planning with Large Language Models for Code Generation

“…However, in reality, human-specified test cases are not always available. Recently, Chen et al (2022) observe that Transformer pre-trained on code generation can also generate useful test cases by adding an assert keyword at the end of the prompt. We follow the prompt design in Chen et al (2022) to automatically generate test cases and run our PG-TD algorithm using the automatically-generated test cases.…”

“…Recently, Chen et al (2022) observe that Transformer pre-trained on code generation can also generate useful test cases by adding an assert keyword at the end of the prompt. We follow the prompt design in Chen et al (2022) to automatically generate test cases and run our PG-TD algorithm using the automatically-generated test cases. Empirically, we confirm that compared with beam search, PG-TD still has a higher strict accuracy by using automaticallygenerated test cases to verify the generated programs.…”

“…When human-specified test cases are not available, our algorithm can still rely on automaticallygenerated test cases. We first follow Chen et al (2022) to use the Codex model and evaluate the code generation algorithms on the HumanEval dataset (Chen et al, 2021a). We adopt the same procedure as Chen et al (2022) to automatically generate test cases.…”

“…We first follow Chen et al (2022) to use the Codex model and evaluate the code generation algorithms on the HumanEval dataset (Chen et al, 2021a). We adopt the same procedure as Chen et al (2022) to automatically generate test cases. Concretely, we use a prompt that includes the natural language description of the problem and an assert statement at the end of the prompt.…”

Prompt-tuned Code Language Model as a Neural Knowledge Base for Type Inference in Statically-Typed Partial Code

“…In-context learning is a novel paradigm that conditions the model on task descriptions and demonstrations to generate answers for the same tasks [25]. It has been applied to various domains, including testing [55], code generation [56], and GUI automation [57]. These works use coarse-grained, direct-inquiry style prompt design.…”