Replication's Role in Software Engineering

Abstract

We provide motivation for researchers to replicate experiments in software engineering. The ideology of replication is discussed. We address the question: Is an experiment worth repeating? The current lack of replication studies is highlighted. We make clear that exact replication is unattainable and we draw on our first experience of performing an external replication. To categorise various kinds of replication, we propose a simple extension to Basili et al.'s framework for experimentation in software engineering. We present guidance as to the level of reported detail required to enable others perform a replication. Our conclusion is that there is only one route for empirical software engineering to follow: to make available laboratory packages of experimental materials to facilitate internal and external replications, especially the latter, which have greater confirming power.

How organisation of architecture documentation affects architectural knowledge retrieval

“…Even though exact replication between experiments in software engineering is unattainable [56], we aimed for maximum consistency between the experiment procedures in the Océ and LaiAn experiments. Several variations between the experiment domains, e.g., those reported in Table 1, indicate that the results are generalizable to other software project domains.…”

Replications of software engineering experiments

“…The few early works are relatively recent (Basili et al 1999;Brooks et al 1995). Although over the last 10 years, there has been some more research aiming at adopting this essential part of the experimental paradigm in SE: (Gómez et al 2010a, b;Shull et al 2002;Vegas et al 2006;Brooks et al 2008;Juristo and Vegas 2009;Krein and Knutson 2010). However, there is no agreement yet on terminology, typology, purposes, operation and other replication issues.…”

A Framework for Software Engineering Experimental Replications

“…An internal replication is conducted by the same set of researchers who conducted the original experiment, while an external replication is conducted by different researchers. Brooks et al provide an excellent discussion about replications in a software engineering environment, highlighting the need for external replications to provide a strong scientific foundation [8].…”

“…Wohlin et al and Juristo and Moreno provide in depth guidelines for performing and reporting controlled experiments [10,29]. Brooks et al, provide guidelines to improve experimental 'recipes', the use of alternative data analysis techniques, and packaging experiments for replication [8].…”