Would wider adoption of reproducible research be beneficial for empirical software engineering research?

Abstract

Abstract.Researchers have identified problems with the validity of software engineering research findings. In particular, it is often impossible to reproduce data analyses, due to lack of raw data, or sufficient summary statistics, or undefined analysis procedures. The aim of this paper is to raise awareness of the problems caused by unreproducible research in software engineering and to discuss the concept of reproducible research (RR) as a mechanism to address these problems. RR is the idea that the outcome of research is both a paper and its computational environment. We report some recent studies that have cast doubts on the reliability of research outcomes in software engineering. Then we discuss the use of RR as a means of addressing these problems. We discuss the use of RR in software engineering research and present the methodology we have used to adopt RR principles. We report a small working example of how to create reproducible research. We summarise advantages of and problems with adopting RR methods. We conclude that RR supports good scientific practice and would help to address some of the problems found in empirical software engineering research.

Evaluating Pred(p) and standardized accuracy criteria in software development effort estimation

“…Internal validity questions the extent to which the association between dependent and independent variables holds . This study investigates the use of Pred( p ) and SA as accuracy measures and model selectors.…”

“…Construct validity observes whether a study measures what it actually intends to measure . The aim of this study was to evaluate and compare the 2 accuracy measures Pred(p) an SA.…”

“…Threats to the conclusion validity are concerned with issues that affect the ability to draw the correct conclusion about relations between the treatment and the outcome of an experiment . A potential limitation may come from the choice of the SDEE techniques.…”

“…Internal validity questions the extent to which the association between dependent and independent variables holds. [48][49][50][51] This study investigates the use of Pred(p) and SA as accuracy measures and model selectors. It should be noted that most internal limitations are a consequence of the scope defined for the study.…”

“…Construct validity observes whether a study measures what it actually intends to measure. [48][49][50][51] The aim of this study was to evaluate and compare the 2 accuracy measures Pred(p) an SA. Given that SA is an unbiased measure, a possible threat may come from the categories we used for Pred(p) and SA (High, Medium, and Low).…”