Measuring Coding Challenge Competence With APPS

Abstract

While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to accurately assess code generation performance rigorously. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an important measure for tracking advancements."Everybody should learn to program a computer, because it teaches you how to think." -Steve Jobs

Investigating Explainability of Generative AI for Code through Scenario-based Design

“…The explainability categories we identified have varied technical feasibility with current techniques, and point to topics that are under-explored for generative AI. For example, for the Performance category, existing works have used the Computational Accuracy metric to evaluate generative code models [9,15,33,88], but not other metrics we uncovered regarding the characteristics of the generated artifacts and run-time efficiency. To understand performance differences and limitations with regard to different types of input, solutions have been explored for natural language generation under Prompt Engineering [57,58].…”

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

“…Large-scale natural language modeling has witnessed rapid advances since the inception of the Transformer architecture [46]. It has been shown by recent works that large language models (LLMs) pre-trained on large unstructured text corpus not only can perform strongly on various down-stream NLP tasks [10,33,34,5] but the learned representations can also be used to model relations of entities [20], retrieve matching visual features [17], synthesize code from docstrings [13,7], solve math problems [8,39], and even as valuable priors when applied to diverse tasks from different modalities [23,45]. Notably, by pre-training on large-scale data, these models can also internalize an implicit knowledge base containing rich information about the world from which factual answers (e.g.…”

Using Document Similarity Methods to create Parallel Datasets for Code Translation

“…To bypass this limitation, Roziere et al (2020) used unsupervised neural machine translation techniques to translate between languages using only monolingual corpora, and showed impressive results for translation between Java, C++, and Python. While Roziere et al (2020) trained the model specifically for code translation, large language models -such as GPT-2 (Radford et al, 2019), GPT-3 (Brown et al, 2020), and Codex -have also been shown to have some competence in generating code (Hendrycks et al, 2021).…”