On the Replicability and Reproducibility of Deep Learning in Software Engineering

Abstract

Deep learning (DL) techniques have gained significant popularity among software engineering (SE) researchers in recent years. This is because they can often solve many SE challenges without enormous manual feature engineering effort and complex domain knowledge. Although many DL studies have reported substantial advantages over other state-of-the-art models on effectiveness, they often ignore two factors: (1) replicability -whether the reported experimental result can be approximately reproduced in high probability with the same DL model and the same data; and (2) reproducibility -whether one reported experimental findings can be reproduced by new experiments with the same experimental protocol and DL model, but different sampled real-world data. Unlike traditional machine learning (ML) models, DL studies commonly overlook these two factors and declare them as minor threats or leave them for future work. This is mainly due to high model complexity with many manually set parameters and the time-consuming optimization process. In this study, we conducted a literature review on 93 DL studies recently published in twenty SE journals or conferences. Our statistics show the urgency of investigating these two factors in SE, where only 10.8% of the studies discussed any research questions affecting replicability and/or reproducibility. More than 74.2% of the studies do not even share source code and data to support the replicability of their complex models. Moreover, we re-ran four representative DL models in SE. Experimental results show the importance of replicability and reproducibility, where the reported performance of a DL model could not be replicated for an unstable optimization process. Reproducibility could be substantially compromised if the model training is not convergent, or if performance is sensitive to the size of vocabulary and testing data. It is therefore urgent for the SE community to provide a long-lasting link to a replication package, enhance DL-based solution stability and convergence, and avoid performance sensitivity on different sampled data.CCS Concepts: • Software and its engineering → Software maintenance tools.

Opportunities and Challenges in Code Search Tools

“…They have also sometimes tuned the parameters without explaining the reasons or validity. Such unverified parameters may threaten the model generalizability [73].…”

“…Due to the above difficulties, most code search studies chose to identify relevancy with human efforts. However, their evaluation with manual identification is subject to potential serious bias and human errors [37,42,73,81,128].…”

“…However, Fig. 4 shows that only 18% of code search tools provide accessible replication packages [73,118]. It is recommended that future studies share their contributions publicly [73].…”

“…4 shows that only 18% of code search tools provide accessible replication packages [73,118]. It is recommended that future studies share their contributions publicly [73]. Some researchers may share their code upon email request, but it is highly recommended to provide the accessible replication package links in the papers and to maintain these repositories.…”

The challenge of reproducible ML: an empirical study on the impact of bugs

“…Pham et al surveyed the literature to understand the state of reproducibility and repeatability in Software Engineering and Artificial Intelligence (AI), and found that only 19.5% of papers tried to use several identical runs when reporting their results [12]. Similarly, a recent analysis of code inclusion in Deep Learning (DL) papers found that 25.8% of them include code [13]. While code inclusion in a research paper goes one step further for reproducibility, it needs to be usable.…”