Outcomes of a community workshop to identify and rank barriers to the systematic literature review process

Hassler, Edgar; Carver, Jeffrey C.; Kraft, Nicholas A.; Hale, David P.

doi:10.1145/2601248.2601274

Cited by 40 publications

(23 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Whilst this has generated feedback identifying possible implementation problems and proposed solutions, it has also aided the review of the initial guidelines proposed by Kitchenham [8,19,37,44,45]. One of the major problems faced by SR users is the amount of time and effort required to conduct a thorough SR [36,42].…”

Section: Introductionmentioning

confidence: 99%

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

Olorisade

Quincey

Brereton

et al. 2016

Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

View full text Add to dashboard Cite

Background: Since the introduction of the systematic review process to Software Engineering in 2004, researchers have investigated a number of ways to mitigate the amount of effort and time taken to filter through large volumes of literature.Aim: This study aims to provide a critical analysis of text mining techniques used to support the citation screening stage of the systematic review process. Method:We critically re-reviewed papers included in a previous systematic review which addressed the use of text mining methods to support the screening of papers for inclusion in a review. The previous review did not provide a detailed analysis of the text mining methods used. We focus on the availability in the papers of information about the text mining methods employed, including the description and explanation of the methods, parameter settings, assessment of the appropriateness of their application given the size and dimensionality of the data used, performance on training, testing and validation data sets, and further information that may support the reproducibility of the included studies.Results: Support Vector Machines (SVM), Naïve Bayes (NB) and Committee of classifiers (Ensemble) are the most used classification algorithms. In all of the studies, features were represented with Bag-of-Words (BOW) using both binary features (28%) and term frequency (66%). Five studies experimented with n-grams with n between 2 and 4, but mostly the unigram was used. χ 2 , information gain and tf-idf were the most commonly used feature selection techniques. Feature extraction was rarely used although LDA and topic modelling were used. Recall, precision, F and AUC were the most used metrics and cross validation was also well used. More than half of the studies used a corpus size of below 1,000 documents for their experiments while corpus size for around 80% of the studies was 3,000 or fewer documents. The major common ground we found for comparing performance assessment based on independent replication of studies was the use of the same dataset but a sound performance comparison could not be established because the studies had little else in common. In most of the studies, insufficient information was reported to enable independent replication. The studies analysed generally did not include any discussion of the statistical appropriateness of the text mining method that they applied. In the case of applications of SVM, none of the studies report the number of support vectors that they found to indicate the complexity of the prediction engine that they use, making it impossible to judge the extent to which over-fitting might account for the good performance results. Conclusions:There is yet to be concrete evidence about the effectiveness of text mining algorithms regarding their use in the automation of citation screening in systematic reviews. The studies indicate that options are still being explored, but there is a need for better reporting as well as more explicit process details and access to datasets to facilitate st...

show abstract

Section: Introductionmentioning

confidence: 99%

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

Olorisade

Quincey

Brereton

et al. 2016

Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

View full text Add to dashboard Cite

show abstract

“…The actual synthesis as such is difficult, but the key problem making it difficult is the primary studies. Two recent papers [5] and [6] present similar findings, Guzman et al [5] identified the primary studies and their quality as the main challenge with respect to synthesis, and Hassler et al [6] report that one of the main barriers for conducting systematic literature studies is the quality of the primary studies.…”

Section: Discussionmentioning

confidence: 67%

“…Hassler et al [6] report on a workshop where experienced researchers discuss some of the challenges with respect to systematic literature reviews. With respect to quality assessment, the highest ranked issues is the quality of the primary studies, which is also the second highest concern overall reported in Hassler et al…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Writing for synthesis of evidence in empirical software engineering

Wohlin

2014

Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

View full text Add to dashboard Cite

Context: Systematic literature reviews have become common in software engineering in the last decade, but challenges remain. Goal: Given the challenges, the objective is to describe improvement areas in writing primary studies, and hence provide a good basis for researchers aiming at synthesizing research evidence in a specific area. Method:The results presented are based on a literature review with respect to synthesis of research results in software engineering with a particular focus on empirical software engineering. The literature review is complemented and exemplified with experiences from conducting systematic literature reviews and working with research methodologies in empirical software engineering. Results: The paper presents three areas where improvements are needed to become more successful in synthesizing empirical evidence. These three areas are: terminology, paper content and reviewing. Conclusion: It is concluded that it must be possible to improve the primary studies, but it requires that researchers start having synthesis in mind when writing their research papers.

show abstract

“…One way of addressing these difficulties is to have extensive SLR planning activities before the conducting phase. It is believed that adequate planning activities is likely to improve the overall performance of the SLR process [7]. Poor planning and inappropriate review strategy will affect the overall quality and efficiency of SLR process.…”

Section: Introductionmentioning

confidence: 99%

Using Suffix Tree Clustering Method To Support The Planning Phase Of Systematic Literature Review

Feng

Chiam

Faiza

et al. 2017

MJCS

View full text Add to dashboard Cite

Systematic Literature Review (SLR) is an information-based process throughout which each stage must be carefully and systematically designed in planning phase. Important decisions need to be made about the various choices involved in the SLR planning phase. However, due to the necessarily comprehensive and rigorous nature of SLR, many researchers have difficulties in planning SLRs sufficiently and effectively. In recent decades, the use of text mining (TM) techniques has been introduced to facilitate SLR process, especially at conducting review phase to support the searching and selection of studies. However, so far, TM techniques have not been applied to support any activities during SLR planning phase. In this research, we proposed a method that apply Suffix Tree Clustering (STC) method to support decision-making activities within the SLR planning phase. This method also known as SLR Planning Based on Suffix Tree Clustering (SLRP-STC) method. It aims to help reviewers in three ways: 1) to extract research topics easily; 2) to better interpret extracted information; and 3) to quickly realize and refine a proposed search string that is poorly formulated or inappropriate. A case study was conducted by comparing the proposed method with manual approach. It is observed that the use of SLRP-STC method can improve the planning of SLR.Keywords: Systematic Literature Review (SLR), planning phase; decision-making, Suffix Tree Clustering (STC), text mining (TM) INTRODUCTIONSystematic Literature Review (SLR), or Systematic Review (SR), provides a fair and reliable approach to identify, evaluate, and synthesize all primary studies, in attempt to address specific research questions [1]. SLR differs from ordinary review process in term of formal planning. Staples and Niazi [2] define a well-planned SLR as "independently replicable and of greater scientific value than ordinary literature reviews". For another, creating a comprehensive review plan is the key to the success of SLR as it drives the entire review methodology. Effective SLR planning takes into consideration all aspects of planning including identification of review need, specification of research questions, development, and validation of the review protocol.Performing SLR comprises three main phases: 1) planning the review, 2) conducting the review, and 3) reporting the review [5]. Particularly, planning phase is important to ensure the overall quality of SLR process. During the planning phase, reviewers are required to have adequate knowledge of the proposed topic area so they are likely to make a strategic plan for the SLR. For example, reviewers should confirm that there are likely to be sufficient relevant studies to make their SLRs worthwhile. They should also obtain the necessary information relevant to the studies, such as the number of independent studies that make review worthwhile, the research topics, the commonly used terms, and keywords. In this case, effective knowledge acquisition and discovery process can be useful and necessary in supporti...

show abstract

Outcomes of a community workshop to identify and rank barriers to the systematic literature review process

Cited by 40 publications

References 21 publications

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

Writing for synthesis of evidence in empirical software engineering

Using Suffix Tree Clustering Method To Support The Planning Phase Of Systematic Literature Review

Contact Info

Product

Resources

About