Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers

Rathbone, John; Hoffmann, Tammy; Glasziou, Paul

doi:10.1186/s13643-015-0067-6

Cited by 144 publications

(166 citation statements)

References 25 publications

Supporting

Mentioning

160

Contrasting

Order By: Relevance

“…Replications in the studies reviewed were often conducted by the same research groups. One such in-team replication led to the creation of ABSTACKR [78,79], a tool developed by Wallace et al, that has been evaluated by another group in [66].…”

Section: Discussionmentioning

confidence: 99%

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

Olorisade

Quincey

Brereton

et al. 2016

Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

View full text Add to dashboard Cite

Background: Since the introduction of the systematic review process to Software Engineering in 2004, researchers have investigated a number of ways to mitigate the amount of effort and time taken to filter through large volumes of literature.Aim: This study aims to provide a critical analysis of text mining techniques used to support the citation screening stage of the systematic review process. Method:We critically re-reviewed papers included in a previous systematic review which addressed the use of text mining methods to support the screening of papers for inclusion in a review. The previous review did not provide a detailed analysis of the text mining methods used. We focus on the availability in the papers of information about the text mining methods employed, including the description and explanation of the methods, parameter settings, assessment of the appropriateness of their application given the size and dimensionality of the data used, performance on training, testing and validation data sets, and further information that may support the reproducibility of the included studies.Results: Support Vector Machines (SVM), Naïve Bayes (NB) and Committee of classifiers (Ensemble) are the most used classification algorithms. In all of the studies, features were represented with Bag-of-Words (BOW) using both binary features (28%) and term frequency (66%). Five studies experimented with n-grams with n between 2 and 4, but mostly the unigram was used. χ 2 , information gain and tf-idf were the most commonly used feature selection techniques. Feature extraction was rarely used although LDA and topic modelling were used. Recall, precision, F and AUC were the most used metrics and cross validation was also well used. More than half of the studies used a corpus size of below 1,000 documents for their experiments while corpus size for around 80% of the studies was 3,000 or fewer documents. The major common ground we found for comparing performance assessment based on independent replication of studies was the use of the same dataset but a sound performance comparison could not be established because the studies had little else in common. In most of the studies, insufficient information was reported to enable independent replication. The studies analysed generally did not include any discussion of the statistical appropriateness of the text mining method that they applied. In the case of applications of SVM, none of the studies report the number of support vectors that they found to indicate the complexity of the prediction engine that they use, making it impossible to judge the extent to which over-fitting might account for the good performance results. Conclusions:There is yet to be concrete evidence about the effectiveness of text mining algorithms regarding their use in the automation of citation screening in systematic reviews. The studies indicate that options are still being explored, but there is a need for better reporting as well as more explicit process details and access to datasets to facilitate st...

show abstract

Section: Discussionmentioning

confidence: 99%

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

Olorisade

Quincey

Brereton

et al. 2016

Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

View full text Add to dashboard Cite

show abstract

“…We used the screening tool ' Abstrackr' to screen the title and then the abstract [42]. We did not use Abstrackr's semi-automated screening function, though it shows promise for substantially reducing the time needed to screen titles and abstracts [43]. At the title and abstract levels, documents were assessed independently, with each reviewer assigned a portion of the literature.…”

Section: Study Inclusion Criteriamentioning

confidence: 99%

Does the gender composition of forest and fishery management groups affect resource governance and conservation outcomes? A systematic map

et al. 2016

View full text Add to dashboard Cite

Background: Women often use natural resources differently than men yet frequently have minimal influence on how local resources are managed. An emerging hypothesis is that empowering more women in local resource decision-making may lead to better resource governance and conservation. Here we focus on the forestry and fisheries sectors to answer the question: What is the evidence that the gender composition of forest and fisheries management groups affects resource governance and conservation outcomes? We present a systematic map detailing the geographic and thematic extent of the evidence base and assessing the quality of the evidence, as per a published a priori protocol. Methods:We screened 11,000+ English-language records in Scopus, CAB abstracts, AGRIS, AGRICOLA, Google Scholar, and Google. The websites of 24 international conservation and development organisations, references of included articles, and relevant systematic reviews were also searched for possible documents. A number of groups and individuals were invited to submit documents through email 'call outs' . The inclusion criteria were that an article refers to women or gender, forests or fisheries, and a resource management group comparison in a non-OECD country plus Mexico and Chile.Results: Seventeen studies met the inclusion criteria. Four were qualitative and 13 were quantitative. Forest studies outnumbered fisheries studies 14-3. The majority of the studies came from India and Nepal and focused on forest management. All 17 studies identified improvements in local natural resource governance, and three identified conservation improvements when women participated in the management of the resources. Only two studies, however, were rated as high quality based on study design. Conclusions: For India and Nepal, there is strong and clear evidence of the importance of including women in forest management groups for better resource governance and conservation outcomes. Outside of India and Nepal, there are substantial gaps in the evidence base, but the South Asian evidence presents a compelling case for extending the research to other geographies to see if similar outcomes exist elsewhere and supports a theory of change linking the participation of women in forestry and fisheries management groups with better resource governance and conservation outcomes.

show abstract

“…Classic systematic review and meta‐analysis textbooks also detail the basics of the abstract screening process; yet, many textbooks lack practical information about best practice methods for this stage such as the use of text‐mining software, pilot training of coders, reconciliation, or managing screeners. Other more recent research focuses on improving the efficiency of screening abstracts by evaluating the effectiveness of text‐mining software . These studies focus on the efficiency of the algorithms used for text‐mining but fail to provide practical advice on how review teams can implement text‐mining software in large‐evidence systematic review projects.…”

Section: Introductionmentioning

confidence: 99%

Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses

Polanin

Pigott

Espelage

et al. 2019

Research Synthesis Methods

205

126

View full text Add to dashboard Cite

screening is one important aspect of conducting a high‐quality and comprehensive systematic review and meta‐analysis. Abstract screening allows the review team to conduct the tedious but vital first step to synthesize the extant literature: winnowing down the overwhelming amalgamation of citations discovered through research databases to the citations that should be “full‐text” screened and eventually included in the review. Although it is a critical process, few guidelines have been put forth since the publications of seminal systematic review textbooks. The purpose of this paper, therefore, is to provide a practical set of best practice guidelines to help future review teams and managers. Each of the 10 proposed guidelines is explained using real‐world examples or illustrations from applications. We also delineate recent experiences where a team of abstract screeners double‐screened 14 923 abstracts in 89 days.

show abstract

Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers

Cited by 144 publications

References 25 publications

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

Does the gender composition of forest and fishery management groups affect resource governance and conservation outcomes? A systematic map

Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses

Contact Info

Product

Resources

About