The role and value of replication in empirical software engineering results

Abstract

Context: Concerns have been raised from many quarters regarding the reliability of empirical research findings and this includes software engineering. Replication has been proposed as an important means of increasing confidence. Objective: We aim to better understand the value of replication studies, the level of confirmation between replication and original studies, what confirmation means in a statistical sense and what factors modify this relationship. Method: We perform a systematic review to identify relevant replication experimental studies in the areas of (i) software project effort prediction and (ii) pair programming. Where sufficient details are provided we compute prediction intervals. Results: Our review locates 28 unique articles that describe replications of 35 original studies that address 75 research questions. Of these 10 are external, 15 internal and 3 internal-same-article replications. The odds ratio of internal to external (conducted by independent researchers) replications of obtaining a 'confirmatory' result is 8.64. We also found incomplete reporting hampered our ability to extract estimates of effect sizes. Where we are able to compute replication prediction intervals these were surprisingly large. Conclusion: We show that there is substantial evidence to suggest that current approaches to empirical replications are highly problematic. There is a consensus that replications are important, but there is a need for better reporting of both original and replicated studies. Given the low power and incomplete reporting of many original studies, it can be unclear the extent to which a replication is confirmatory and to what extent it yields additional knowledge to the software engineering community. We recommend attention is switched from replication research to meta-analysis.

Replication of Studies in Empirical Software Engineering: A Systematic Mapping Study, From 2013 to 2018

“…They concluded that internal replications find confirmatory evidence eight times more than external replications. They also stressed the need to improve reporting on the original, replication and expected results [12].…”

Comparing techniques for aggregating interrelated replications in software engineering

“…As discussed by de Magalhaes et al [51] and Shepperd et al [71], one key issue related to replication in SE addresses the different procedures that can be followed to increase the chance of running a successful replication. Shepperd et al [71] highlight that this issue has been addressed by only a few studies and superficially. In a systematic mapping study that we conducted [68], we identified the techniques used to aggregate the results of groups of replications in SE: p-values, and more recently, effect sizes and their 95% confidence intervals (95% CIs), prediction intervals (95% PIs) and meta-analysis [79] [41].…”

“…Mature experimental disciplines, such as medicine, recommend the use of meta-analysis [33] [8] [78] [17] [7]. Some authors have also proposed its use in SE [71]. To the best of our knowledge, however, there has been no previous assessment in SE of how these approaches perform.…”

A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications

“…Ever since, meta-analysis has been tightly coupled in SE with the concept of synthesizing the results of already published studies (i.e., typically with standardized effect sizes such as Cohen's d and AD) [3], [101], [129]- [131]. To do this, researchers should backcalculate appropriate effect sizes from study reports, and, if the studies are not very dissimilar, use either fixed-effects models or random-effects models for combination [3], [100], [101], [130].…”