Use and Misuse of the Term “Experiment” in Mining Software Repositories Research

Abstract

The significant momentum and importance of Mining Software Repositories (MSR) in Software Engineering (SE) has fostered new opportunities and challenges for extensive empirical research. However, MSR researchers seem to struggle to characterize the empirical methods they use into the existing empirical SE body of knowledge. This is especially the case of MSR experiments. To provide evidence on the special characteristics of MSR experiments and their differences with experiments traditionally acknowledged in SE so far, we elicited the hallmarks that differentiate an experiment from other types of empirical studies and characterized the hallmarks and types of experiments in MSR. We analyzed MSR literature obtained from a smallscale systematic mapping study to assess the use of the term experiment in MSR. We found that 19% of the papers claiming to be an experiment are indeed not an experiment at all but also observational studies, so they use the term in a misleading way. From the remaining 81% of the papers, only one of them refers to a genuine controlled experiment while the others stand for experiments with limited control. MSR researchers tend to overlook such limitations, compromising the interpretation of the results of their studies. We provide recommendations and insights to support the improvement of MSR experiments.

Code Comment Analysis–A Review Paper

“…Just the consistency category is discussed in this study. Consistency and utility may be found in Ayala (2021).…”

Anchoring code understandability evaluations through task descriptions

“…We now take this one step further and describe on a conceptual level what code comprehension can mean in the context of empirical studies. Thereby, we restrict ourselves to empirical studies with an experimental character 3 to ensure a certain comparability of the study designs. And we remain at a level of abstraction that allows us to map the majority of existing code comprehension experiments to the proposed conceptual model.…”

“…We are aware that the term experiment is used differently[58] or even incorrectly[3] in software engineering research, and that there are different types of experiments. The experiments we focus on here are closest to what Stol and Fitzgerald classify as 'Laboraty Experiment', the purpose of which is "to study with a high degree of precision relationships between variables, or comparisons between techniques; may allow establishment of causality between variables"[58].…”