Problems with Statistical Practice in Human-Centric Software Engineering Experiments

Abstract

Background Examples of questionable statistical practice, when published in high quality software engineering (SE) journals, may lead to novice researchers adopting incorrect statistical practices. Objective Our goal is to highlight issues contributing to poor statistical practice in human-centric SE experiments. Method We reviewed the statistical analysis practices used in the 13 papers that reported families of human-centric SE experiments and were published in high quality journals. Results Reviewed papers related to 45 experiments and involved a total of 1303 human participants. We searched for issues that were related to questionable statistical practice that were found in more than one paper. We observed three types of bad practice: incorrect use of terminology, incorrect analysis of repeated measures designs, and post-hoc power testing. We also found two analysis practices (i.e., multiple testing and pre-testing for normality) where statisticians disagree about good practice. Conclusions Identified issues pose a problem because readers may expect the statistical methods used in papers published in top quality, peer-reviewed journals to be correct. We explain why the practices are problematic and provide recommendations for improved practice. CCS CONCEPTS • General and reference → Empirical studies; Experimentation; • Software and its engineering → Software creation and management.

Meta-analysis for families of experiments in software engineering: a systematic review and reproducibility and validity assessment

“…During data extraction, it became clear that many of our 13 primary studies, included experiments with crossover designs. Vegas et al (2016) warned that the terminology used to describe crossover designs was not used consistently, and we found exactly the same problem with our primary studies (Kitchenham et al 2019a). Therefore, we used the description of the experimental design provided by the authors to derive our own classification.…”

Publication Bias

“…Finally, we also have observed some weaknesses in the experimenters' statistical knowledge. This problem was pointed out by other researchers, e.g., [17,28] previously. The ESEM community (and the overall SE community, as well) should establish measures to improve experimenters' statistical skills.…”

Evolution of statistical analysis in empirical software engineering research: Current state and steps forward

“…Given our focus on journals, we extracted data from: Transactions on Software Engineering (TSE), Transactions on Software Engineering and Methodology (TOSEM), Empirical Software Engineering (EMSE), Journal of Systems and Software (JSS), and Information and Software Technology (IST). The same sample of journals was used in previous studies by Kitchenham et al [52]. The main reason for selecting these five journals is that they are well-known and top-ranked software engineering journals focusing primarily on applied scientific contributions.…”

“…Even though they are a subset of existing peer-reviewed publication venues, they comprise five popular and top-ranked SE journals. Also, the same sample has been used in similar ESE studies [52]. An alternative would have been to include conference papers as well, but we argue that the limited number of pages available to papers in conferences could hinder researchers to report thoroughly on their empirical studies, e.g., not including enough details on the choice and usage of statistics.…”

“…Some studies focus on certain aspects of the experimental process, e.g., subject experience [89,43], bias [87], replication [48], or reporting [56,46,47]. The early work on guidelines on experimentation by Kitchenham et al [54], and the corresponding work on how to evaluate them [51], is especially relevant as researchers refine those guidelines as new studies are performed [95,52,89].…”