A survey of controlled experiments in software engineering

Abstract

Background: Like any research discipline, software engineering research must be of a certain quality to be valuable. High quality research in software engineering ensures that knowledge is accumulated and helpful advice is given to the industry. One way of assessing research quality is to conduct systematic reviews of the published research literature. Objective: The purpose of this work was to assess the quality of published experiments in software engineering with respect to the validity of inference and the quality of reporting. More specifically, the aim was to investigate the level of statistical power, the analysis of effect size, the handling of selection bias in quasi-experiments, and the completeness and consistency of the reporting of information regarding subjects, experimental settings, design, analysis, and validity. Furthermore, the work aimed at providing suggestions for improvements, using the potential deficiencies detected as a basis. Method: The quality was assessed by conducting a systematic review of the 113 experiments published in nine major software engineering journals and three conference proceedings in the decade 1993-2002. Results: The review revealed that software engineering experiments were generally designed with unacceptably low power and that inadequate attention was paid to issues of statistical power. Effect sizes were sparsely reported and not interpreted with respect to their practical importance for the particular context. There seemed to be little awareness of the importance of controlling for selection bias in quasi-experiments. Moreover, the review revealed a need for more complete and standardized reporting of information, which is Sample size 113 113 100 Mortality rate 113 21.2 Type (student/professionals) 112 113 99.1 Recruitment (Voluntarily/mandatory) 113 36.3 Some kind of background information 113 87.6-Programming experience 113 32.7-Work experience 113 21.2-Task related experience 113 70.8 Subjects-Grades 6 113 5.3 Task 113 113 100.0 Duration 113 61.1 Application system 101 113 89.4 Size of materials 113 59.3 Location 113 35.4 Experimental setting The use of tools 113 54.9

Comparing OpenBRR, QSOS, and OMM Assessment Models

“…Our plan was to conduct a controlled experiment [5,4]. We managed to satisfy many requirements for a controlled experiment as: randomization of participants, and testing specimens, and the set up of a controlled environment, the detailed planning of the experimentation process, and others.…”

Ontology-driven conceptual modeling: A systematic literature mapping and review

“…The main threats to the validity of a SLR are (1) publication selection bias, (2) inaccuracy in data extraction and (3) misclassification (Sjøberg et al, 2005). We acknowledge that is it impossible to achieve complete coverage of everything written on a topic.…”

Does the Level of Detail of UML Models Affect the Maintainability of Source Code?

“…Working with students also implies various advantages, such as the fact that their prior knowledge is fairly homogeneous, there is the possible availability of a large number of subjects [15], and there is the chance to test experimental design and initial hypotheses [16]. An additional advantage of using novices as subjects in experiments on comprehensibility and modifiability is that the cognitive complexity of the objects under study is not hidden by the subjects' experience.…”