Background: Like any research discipline, software engineering research must be of a certain quality to be valuable. High quality research in software engineering ensures that knowledge is accumulated and helpful advice is given to the industry. One way of assessing research quality is to conduct systematic reviews of the published research literature. Objective: The purpose of this work was to assess the quality of published experiments in software engineering with respect to the validity of inference and the quality of reporting. More specifically, the aim was to investigate the level of statistical power, the analysis of effect size, the handling of selection bias in quasi-experiments, and the completeness and consistency of the reporting of information regarding subjects, experimental settings, design, analysis, and validity. Furthermore, the work aimed at providing suggestions for improvements, using the potential deficiencies detected as a basis. Method: The quality was assessed by conducting a systematic review of the 113 experiments published in nine major software engineering journals and three conference proceedings in the decade 1993-2002. Results: The review revealed that software engineering experiments were generally designed with unacceptably low power and that inadequate attention was paid to issues of statistical power. Effect sizes were sparsely reported and not interpreted with respect to their practical importance for the particular context. There seemed to be little awareness of the importance of controlling for selection bias in quasi-experiments. Moreover, the review revealed a need for more complete and standardized reporting of information, which is Sample size 113 113 100 Mortality rate 113 21.2 Type (student/professionals) 112 113 99.1 Recruitment (Voluntarily/mandatory) 113 36.3 Some kind of background information 113 87.6-Programming experience 113 32.7-Work experience 113 21.2-Task related experience 113 70.8 Subjects-Grades 6 113 5.3 Task 113 113 100.0 Duration 113 61.1 Application system 101 113 89.4 Size of materials 113 59.3 Location 113 35.4 Experimental setting The use of tools 113 54.9

No abstract

No abstract