Comparing Approaches for Quality Evaluation of Software Engineering Experiments: An Empirical Study on Software Product Line Experiments

Abstract

The Software Engineering (SE) research area must provide results of a certain quality for the sake of value. High quality research results may ensure experience and knowledge, which are essential for the technology to be transferred to the industry. One of the means to obtain such quality results is experimentation. Experimentation is a scientific method that aims to provide evidence of a theory over real-world observations establishing a cause-effect relation. Well conducted, auditable and repeatable experiments are vital for scientific evolution and novelty. Quality evaluation of controlled experiments and quasi-experiments in SE has been recently discussed in the literature as researchers desire to assess whether such experiments have improved by reporting information that enables the experiments to be replicated and the reader can understand the experiment and validate results. Thus, this work empirically compares four approaches for quality evaluation of SE experiments in the context of Software Product Lines (SPL). In addition, we are interested on verifying the quality of reporting experiments in a well-discussed reuse technique as SPL. The Pearson technique supported the correlation between pairs of evaluation approaches. In addition, the T-Test and Mann-Whitney-Wilcoxon U test were applied to the samples to verify whether there was a difference in the quality of experiments when using an experimental template. Preliminary results show a strong positive correlation between them, the hypothesis tests confirmed there is such a difference in quality when using experimental template and the SPL experiments report more the planning phase than the analysis and interpretation phase. Based on our results, we provide initial evidence two approaches are the best to reporting SPL experiments.