A Map of Threats to Validity of Systematic Literature Reviews in Software Engineering

Abstract

Context: The assessment of Threats to Validity (TTVs) is critical to secure the quality of empirical studies in Software Engineering (SE). In the recent decade, Systematic Literature Review (SLR) was becoming an increasingly important empirical research method in SE. One of the mechanisms of insuring the level of scientific value in the findings of an SLR is to rigorously assess its validity. Hence, it is necessary to realize the status quo and issues of TTVs of SLRs in SE. Objective: This study aims to investigate thestate-of-the-practice of TTVs of the SLRs published in SE, and further support SE researchers to improve the assessment and strategies against TTVs in order to increase the quality of SLRs in SE. Method: We conducted a tertiary study by reviewing the SLRs in SE that report the assessment of TTVs. Results: We identified 316 SLRs published from 2004 to the first half of 2015, in which TTVs are discussed. The issues associated to TTVs were also summarized and categorized. Conclusion: The common TTVs related to SLR research, such as internal validity and reliability, were thoroughly discussed in most SLRs. The threats to construct validity and external validity drew less attention. Moreover, there are few strategies and tactics being reported to cope with the various TTVs.

Usability Measures in Mobile-Based Augmented Reality Learning Applications: A Systematic Review

“…A study by [7] highlighted four major threats to validity (TTV) in SLRs, and this research acknowledges these shortcomings by strategizing to minimize the risks of these TTVs. The four threats of validity were resolved in four separate strategies:…”

“…Each optional answer was given a coded point of either "Yes" = 1, "Partly" = 0.5, or "No" = 0. As per the practice of [7,11], QAs were done meticulously where results of the assessments were discussed by the authors prior to approval. From this process, a total of 72 papers scored 2.5 or more (50% or more), and these were selected to be included in the study of this research.…”

“…Internal validity was solved by adopting a method used by [7]. In order to eliminate biases in paper selection through an exhaustive search, a combination approach of automated search and snowballing was carried out for a more inclusive selection approach.…”

Studying Software Engineering Patterns for Designing Machine Learning Systems

“…Finally, we could draw the wrong conclusion from the survey answers but backed all our answers by the data. b) SLR: SLRs can also [47] have threats to the validity of their results: mostly internal validity and reliability. Internal validity concerns the cause-effect conclusion that we drew from the SLR process and its results.…”

A Systematic Literature Review of Applications of the Physics of Notations

“…According to Kitchenham, the most important threats to validity are those that introduce bias into the review process [29]. Zhou et al further detail the most discussed and prevalent threats to validity of SLRs [44], which we address for this SLR here:…”