No abstract

Background:] Experimentation in Software Engineering plays a central role on sharing and verifying scientific findings. As experiments have increased significantly in Software Engineering area, we observe that most of them fail to provide a way to be repeated, replicated or reproduced, thus jeopardizing or delaying the evolution of the Software Engineering area. [Aims:] In this vision paper, we present and discuss techniques and infrastructure to continuously improve experiments towards repeatability, replicability, and reproducibility. [Method:] We define these techniques and infrastructure based on experiences of our research groups and existing literature. Furthermore, we follow Open Science principles. [Results:] We provide incipient results and foresee a central infrastructure composed of two repositories and two recommendation systems to support techniques for: reporting experiments; developing ontologies for experiments and open educational resources; mining and recommending experiments; specifying data management plans, identifying families of experiments; and teaching and learning experimentation. [Conclusions:] Our techniques and infrastructure will prospectively motivate and benefit Software Engineering evolution by improving the conduction and further reproducibility of experiments. CCS CONCEPTS• Software and its engineering → Empirical software validation;

The Software Engineering (SE) research area must provide results of a certain quality for the sake of value. High quality research results may ensure experience and knowledge, which are essential for the technology to be transferred to the industry. One of the means to obtain such quality results is experimentation. Experimentation is a scientific method that aims to provide evidence of a theory over real-world observations establishing a cause-effect relation. Well conducted, auditable and repeatable experiments are vital for scientific evolution and novelty. Quality evaluation of controlled experiments and quasi-experiments in SE has been recently discussed in the literature as researchers desire to assess whether such experiments have improved by reporting information that enables the experiments to be replicated and the reader can understand the experiment and validate results. Thus, this work empirically compares four approaches for quality evaluation of SE experiments in the context of Software Product Lines (SPL). In addition, we are interested on verifying the quality of reporting experiments in a well-discussed reuse technique as SPL. The Pearson technique supported the correlation between pairs of evaluation approaches. In addition, the T-Test and Mann-Whitney-Wilcoxon U test were applied to the samples to verify whether there was a difference in the quality of experiments when using an experimental template. Preliminary results show a strong positive correlation between them, the hypothesis tests confirmed there is such a difference in quality when using experimental template and the SPL experiments report more the planning phase than the analysis and interpretation phase. Based on our results, we provide initial evidence two approaches are the best to reporting SPL experiments.

No abstract

No abstract