A systematic review of systematic review process research in software engineering

Abstract

Context: Many researchers adopting systematic reviews (SRs) have also published papers discussing problems with the SR methodology and suggestions for improving it. Since guidelines for SRs in software engineering (SE) were last updated in 2007, we believe it is time to investigate whether the guidelines need to be amended in the light of recent research. Objective: To identify, evaluate and synthesize research published by software engineering researchers concerning their experiences of performing SRs and their proposals for improving the SR process. Method: We undertook a systematic review of papers reporting experiences of undertaking SRs and/or discussing techniques that could be used to improve the SR process. Studies were classified with respect to the stage in the SR process they addressed, whether they related to education or problems faced by novices and whether they proposed the use of textual analysis tools. Results: We identified 68 papers reporting 63 unique studies published in SE conferences and journals between 2005 and mid-2012. The most common criticisms of SRs were that they take a long time, that SE digital libraries are not appropriate for broad literature searches and that assessing the quality of empirical studies of different types is difficult. Conclusion:We recommend removing advice to use structured questions to construct search strings and including advice to use a quasi-gold standard based on a limited manual search to assist the construction of search stings and evaluation of the search process. Textual analysis tools are likely to be useful for inclusion/exclusion decisions and search string construction but require more stringent evaluation. SE researchers would benefit from tools to manage the SR process but existing tools need independent validation. Quality assessment of studies using a variety of empirical methods remains a major problem.

A survey on business processes management suites

“…This paper shows a formal survey which combines well-known techniques such as SLR [23] and quality frameworks [7]. Thanks to this paper, organizations can formally plan, manage and perform a comparative on BPMSs and, later, analyze achieved results.…”

“…Taking into account all these proposals, we have decided to follow the protocol defined by Kitchenham in our survey [ [21][22][23]] because it is considered one of the most acknowledged in software engineering. Our paper also takes into consideration Wohlin and Prikladnicki's [51] conclusions about SLRs in software engineering, since they consider that the search strategy is key to ensure a good starting point for the identification of studies and ultimately for the actual outcome of a particular study.…”

“…This paper aims to describe a survey with which organizations can compare specific BPMS according to their own organizational objectives. This survey is oriented to each phase of the process lifecycle and it has been carried out using a proven method [8] which combines well-known techniques such as Systematic Literature Review (SLR) [23] and quality frameworks [7] (based on a characterization scheme). This technique used in our survey has already been used in other contexts [9] and we have obtained successful results thanks to the application of the quality models.…”

Practical similarities and differences between Systematic Literature Reviews and Systematic Mappings: a tertiary study

“…The main contribution of our paper is to establish a fair and practical discussion about the conduction of secondary studies in SE area. Several studies have reported experiences and lessons learned from researchers conducting secondary studies [3], [5], [6], [7]. However, these studies do not focus on the practical similarities and differences between secondary studies.…”

Protocol adaptations to conduct Systematic Literature Reviews in Software Engineering: a chronological study

“…Therefore, SLR protocol adaptations to SE must be considered. Additionally, in a study from Kitchenham (2013) regarding using SLRs in SE, she concludes that the three most significant problems are: (i) digital libraries are not well suited to complex automated searches, (ii) the time and effort needed for SLRs and (iii) the quality assessment of papers based on different research methods.…”