PyMT5: multi-mode translation of natural language and Python code with transformers

Abstract

Simultaneously modeling source code and natural language has many exciting applications in automated software development and understanding. Pursuant to achieving such technology, we introduce PYMT5, the PYTHON method text-to-text transfer transformer, which is trained to translate between all pairs of PYTHON method feature combinations: a single model that can both predict whole methods from natural language documentation strings (docstrings) and summarize code into docstrings of any common style. We present an analysis and modeling effort of a large-scale parallel corpus of 26 million PYTHON methods and 7.7 million method-docstring pairs, demonstrating that for docstring and method generation, PYMT5 outperforms similarlysized auto-regressive language models (GPT2) which were English pre-trained or randomly initialized. On the CODE-SEARCHNET test set, our best model predicts 92.1% syntactically correct method bodies, achieved a BLEU score of 8.59 for method generation and 16.3 for docstring * Corresponding author † Work done during a Microsoft internship generation (summarization), and achieved a ROUGE-L F-score of 24.8 for method generation and 36.7 for docstring generation.

Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy

“…The ROUGE-L metrics are dramatically improved, and is not necessarily surprising as XPyMT5 is conditioned on much more information than PyMT5. The syntax correctness of our fine-tuned models is slightly lower than the 92.1% reported by Clement et al (2020).…”

“…eWASH yields N total training samples from a file with N total methods and class methods. For docstring completion or code summarization, the source contains the method signature and body, and the target contains the desired docstring, and a control code is used to instruct the model which task it is to perform, just like PyMT5 (Clement et al, 2020).…”

“…XPyMT5 uses the same whitespace-augmented GPT-2 (Radford et al, 2018) tokenizer as PyMT5, with about a vocabulary size of 50,000, and is the same architecture and hyperparameters as PyMT5 with 12 layers and 406M parameters. XPyMT5 was trained on 16 32GB Tesla V100 GPUs for 4 weeks, about 10 epochs total, using the same hyperparameters as reported by Clement et al (2020). XPyMT5 was initialized with the English pretrained BART (Lewis et al, 2019) weights (with whitespace embeddings) and pre-trained using the BART de-noising objective for 5 weeks on the same hardware as above.…”

“…Large transformer models (Vaswani et al, 2017) and the pre-training/fine-tuning paradigm (Devlin et al, 2018;Lewis et al, 2019;Radford et al, 2018) have become an essential part of state of the art natural language processing. Beyond the domain of natural language, these models and procedures have enabled rapid progress in the software engineering space, including applications in code completion (Svyatkovskiy et al, , 2019Clement et al, 2020;Raychev et al, 2014;Bruch et al, 2009), natural language to code (NL2Code), code feature summarization (Clement et al, 2020;Moreno et al, 2013;Scalabrino et al, 2017;Wan et al, 2018;Alon et al, 2018;Moreno et al, 2014), code search (Husain et al, 2019;Feng et al, 2020), unit test generation (Tufano et al, 2020) and even bug fixing and detection (Zhai et al, 2020).…”

Unified Pre-training for Program Understanding and Generation

“…Code Summarization (Movshovitz-Attias and Cohen, 2013;Allamanis et al, 2016;Iyer et al, 2016;Alon et al, 2019a;Hu et al, 2018;Harer et al, 2019;Ahmad et al, 2020), Bug Detection (Ray et al, 2016;Li et al, 2018b;Russell et al, 2018;, Program Repair (Chen et al, 2019;Lutellier et al, 2020), Code Translation (Chen et al, 2018;Drissi et al, 2018;Xu et al, 2020), Clone Detection (Zhang et al, 2019;Yu et al, 2019;, Code completion (Li et al, 2018a;Hellendoorn and Devanbu, 2017;Parvez et al, 2018) are some of the tasks that are addressed with deep neural solution. While most of the prior approaches use task-specific representation learning, a few works (Alon et al, 2019b;Feng et al, 2020;Lachaux et al, 2020;Clement et al, 2020) attempted to learn transferable representations in an unsupervised fashion. More closely to our work, CodeBERT (Feng et al, 2020) is pre-trained on bimodal data to capture the semantic interaction between the input modalities (i.e., program and natural languages).…”