Multi-lingual Evaluation of Code Generation Models

Athiwaratkun, Ben; Gouda, Sanjay Krishna; Wang, Zijian; Li, Xiaopeng; Tian, Ye; Tan, Ming; Ahmad, Wasi Uddin; Wang, Shiqi; Sun, Qizhen; Shang, Mingyue; Gonugondla, Sujan K.; Ding, Hantian; Kumar, Vijay; Fulton, Nathan; Farahani, Arash; Jain, Siddhartha; Giaquinto, Robert; Qian, Haifeng; Ramanathan, Murali; Nallapati, Ramesh; Ray, Baishakhi; Bhatia, Parminder; Sengupta, Sudipta; Roth, Dan; Xiang, Bing

doi:10.48550/arxiv.2210.14868

Cited by 5 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These primarily focus on natural language to Python code generation: HumanEval (Chen et al, 2021), HumanEval+ (Liu et al, 2023b), APPS (Hendrycks et al, 2021), Code-Contests , MBPP , L2CEval (Ni et al, 2023). Their variants have been proposed to cover more languages, (Wang et al, 2022a;Cassano et al, 2022;Athiwaratkun et al, 2022). Many benchmarks have focused on code generation in APIs.…”

Section: Code Generationmentioning

confidence: 99%

Contrastive Code Representation Learning

Jain¹,

Jain²,

Zhang³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Recent work learns contextual representations of source code by reconstructing tokens from their context. For downstream semantic understanding tasks like code clone detection, these representations should ideally capture program functionality. However, we show that the popular reconstruction-based RoBERTa model is sensitive to source code edits, even when the edits preserve semantics. We propose Con-traCode: a contrastive pre-training task that learns code functionality, not form. Con-traCode pre-trains a neural network to identify functionally similar variants of a program among many non-equivalent distractors. We scalably generate these variants using an automated source-to-source compiler as a form of data augmentation. Contrastive pretraining outperforms RoBERTa on an adversarial code clone detection benchmark by 39% AUROC. Surprisingly, improved adversarial robustness translates to better accuracy over natural code; ContraCode improves summarization and TypeScript type inference accuracy by 2 to 13 percentage points over competitive baselines. All source is available at https://github.com/parasj/contracode.

show abstract

Section: Code Generationmentioning

confidence: 99%

Contrastive Code Representation Learning

Jain¹,

Jain²,

Zhang³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…BabelCode shares many design similarities to the concurrent work from Athiwaratkun et al (2022). Specifically, we follow the same approach to inferring argument and return types.…”

Section: Framework Designmentioning

confidence: 99%

“…We summarize the high-level differences between Babel-Code and prior works in Table 1. The MBXP framework from Athiwaratkun et al (2022) is the most similar to our work as discussed in subsection 2.1. Similar to BabelCode, MBXP does have individual test-case results; however, it uses assert statements and thus can only determine the first test-case that fails.…”

Section: Differences To Prior Workmentioning

confidence: 99%

“…Our presented framework is closely related to the concurrent work of MBXP (Athiwaratkun et al, 2022) and Multi-PLE (Cassano et al, 2022). While MBXP is quite similar to BabelCode, it is not open-sourced and requires that the input benchmarks be in Python.…”

Section: Introductionmentioning

confidence: 98%

“…Additional works have introduced generation(Li et al, 2022) and translation(Roziere et al, 2020) tasks in multiple-languages, but are limited to only C++, Java, and Python. We acknowledge concurrent works byCassano et al (2022) andAthiwaratkun et al (2022) on translating HumanEval and MBPP into multiple programming languages. As we note in subsection 2.2, BabelCode supports deeper analysis on a wider range of tasks while including significant methods for ensuring correctness.Code LLMs Recent years has seen significant interest in LLMs for code.CodeBERT (Feng et al, 2020) is the first work to train an encoder only model on code.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation

Orlanski¹,

Gittens²

2021

Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021)

View full text Add to dashboard Cite

Answering a programming question using only its title is difficult as salient contextual information is omitted. Based on this observation, we present a corpus of over 40,000 StackOverflow question texts to be used in conjunction with their corresponding intents from the CoNaLa dataset . Using both the intent and question body, we use BART to establish a baseline BLEU score of 34.35 for this new task. We find further improvements of 2.8% by combining the mined CoNaLa data with the labeled data to achieve a 35.32 BLEU score. We evaluate prior stateof-the-art CoNaLa models with this additional data and find that our proposed method of using the body and mined data beats the BLEU score of the prior state-of-the-art by 71.96%. Finally, we perform ablations to demonstrate that BART is an unsupervised multimodal learner and examine its extractive behavior. 1

show abstract