Statistical machine translation outperforms neural machine translation in software engineering: why and how

Abstract

Neural Machine Translation (NMT) is the current trend approach in Natural Language Processing (NLP) to solve the problem of automatically inferring the content of target language given the source language. The ability of NMT is to learn deep knowledge inside languages by deep learning approaches. However, prior works show that NMT has its own drawbacks in NLP and in some research problems of Software Engineering (SE). In this work, we provide a hypothesis that SE corpus has inherent characteristics that NMT will confront challenges compared to the state-of-the-art translation engine based on Statistical Machine Translation. We introduce a problem which is significant in SE and has characteristics that challenges the ability of NMT to learn correct sequences, called Prefix Mapping. We implement and optimize the original SMT and NMT to mitigate those challenges. By the evaluation, we show that SMT outperforms NMT for this research problem, which provides potential directions to optimize the current NMT engines for specific classes of parallel corpus. By achieving the accuracy from 65% to 90% for code tokens generation of 1000 Github code corpus, we show the potential of using MT for code completion at token level.

Can We Generate Shellcodes via Natural Language? An Empirical Study

“…NMT models: Before the era of NMT, Statistical Machine Translation (SMT) [15] was the most popular technique for software engineering (SE) problems, it still outperforms NMT in some SE problems [71]. However, since we are interested in the specific problem of code generation, we focus on NMT that has shown superior performance on public benchmarks [9], and that it is widely recognized as the premier method for the translation of different languages [83].…”

Can we generate shellcodes via natural language? An empirical study

“…NMT models Before the era of NMT, Statistical Machine Translation (SMT) Costa-Jussá and Farrús (2014) was the most popular technique for software engineering (SE) problems, it still outperforms NMT in some SE problems (Phan and Jannesari 2020). However, since we are interested in the specific problem of code generation, we focus on NMT that has shown superior performance on public benchmarks (Bojar et al 2016), and that it is widely recognized as the premier method for the translation of different languages (Wu et al 2016).…”

A Survey on Machine Learning Techniques for Source Code Analysis

“…Li et al [180] conducted experiments on two datasets to demonstrate the effectiveness of their approach consisting of an attention mechanism and a pointer mixture network on code completion tasks. Phan and Jannesari [245] used three corpus for their experiments-a large-scale corpus of English-German translation in nlp [201], the Conala corpus [356], which contains Python software documentation as 116,000 English sentences, and the msr 2013 corpus [18]. Schuster et al [275] used a public archive of GitHub from 2020 [1].…”

“…Gopalakrishnan et al [109] extracted relationships between topical concepts in the source code and the use of specific architectural developer tactics in that code. Phan and Jannesari [245] used machine translation to learn the mapping from prefixes to code tokens for code suggestion. They extracted the tokens from the documentation of the source code.…”

“…Chen and Wan [70] applied Tree2Seq model with nmt techniques for code comment generation. Phan and Jannesari [245] proposed PrefixMap, a code suggestion tool for all types of code tokens in the Java programming language. Their approach uses statistical machine translation that outperforms nmt.…”