Learning to mine aligned code and natural language pairs from stack overflow

Abstract

For tasks like code synthesis from natural language, code retrieval, and code summarization, data-driven models have shown great promise. However, creating these models require parallel data between natural language (NL) and code with fine-grained alignments. Stack Overflow (SO) is a promising source to create such a data set: the questions are diverse and most of them have corresponding answers with high quality code snippets. However, existing heuristic methods (e.g., pairing the title of a post with the code in the accepted answer) are limited both in their coverage and the correctness of the NL-code pairs obtained. In this paper, we propose a novel method to mine high-quality aligned data from SO using two sets of features: hand-crafted features considering the structure of the extracted snippets, and correspondence features obtained by training a probabilistic model to capture the correlation between NL and code using neural networks. These features are fed into a classifier that determines the quality of mined NL-code pairs. Experiments using Python and Java as test beds show that the proposed method greatly expands coverage and accuracy over existing mining methods, even when using only a small number of labeled examples. Further, we find that reasonable results are achieved even when training the classifier on one language and testing on another, showing promise for scaling NL-code mining to a wide variety of programming languages beyond those for which we are able to annotate data.

JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation

“…Although various large scale datasets Sutton, 2013, 2014;Allamanis et al, 2016; to study code generation have been created from Github, their development and test set are randomly created from the same dataset since human curation is prohibitively expensive. Similarly, Yin et al (2018) collect a large dataset from Stackoverflow.com (CoNaLa) for training, but only manage to curate a small portion (∼ 2,900 examples) of single line NL and code snippets for evaluation. We take advantage of nbgrader assignment notebooks to create an inexpensive highquality human-curated test set of 3,725 NL statements with interactive history.…”

“…Existing tasks for mapping NL to source code primarily use a single NL utterance (Zettlemoyer and Collins, 2005;Iyer et al, 2017) to generate database queries (semantic parsing), single line python code (Yin et al, 2018;Oda et al, 2015), multi-line domain-specific code (Ling et al, 2016;Rabinovich et al, 2017), or sequences of API calls (Gu et al, 2016b). A recent task by on the CONCODE dataset maps a single utterance to an entire method, conditioned on environment variables and methods.…”

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

“…It uses the Floating Parser architecture, which is a grammar-based approach that provides more flexibility without requiring hand-engineering of lexicalized rules like synchronous CFG or CCG based semantic parsers [42]. This approach also provides more interpretable results and requires less training data than neural network approaches (e.g., [51,52]). The parser parses user utterances into expressions in a simple functional DSL we created for PUMICE.…”

Jointly Learning Semantic Parser and Natural Language Generator via Dual Information Maximization

“…11.9 y self. max entries = int(max entries) 8.2 CONALA x more pythonic alternative for getting a value in range not using min and max 9.7 y a = 1 if x < 1 else 10 if x > 10 else x 14.1 Figure 2: Sample natural language utterances and meaning representations from datasets used in this work: ATIS for dialogue management; DJANGO (Oda et al, 2015) and CONALA (Yin et al, 2018a) for code generation and summarization.…”

“…is reported 62.3 SNM (Yin and Neubig, 2017) 71.6 COARSE2FINE (Dong and Lapata, 2018) 74 (Hu et al, 2018) 65.9 for parser evaluation based on exact match, and BLEU-4 is adopted for generator evaluation. For the code generation task in CONALA, we use BLEU-4 following the setup in Yin et al (2018a).…”