Screening nonrandomized studies for medical systematic reviews: A comparative study of classifiers

Abstract

Objectives To investigate whether (1) machine learning classifiers can help identify nonrandomized studies eligible for full-text screening by systematic reviewers; (2) classifier performance varies with optimization; and (3) the number of citations to screen can be reduced. Methods We used an open-source, data-mining suite to process and classify biomedical citations that point to mostly nonrandomized studies from 2 systematic reviews. We built training and test sets for citation portions and compared classifier performance by considering the value of indexing, various feature sets, and optimization. We conducted our experiments in 2 phases. The design of phase I with no optimization was: 4 classifiers × 3 feature sets × 3 citation portions. Classifiers included k-nearest neighbor, naïve Bayes, complement naïve Bayes, and evolutionary support vector machine. Feature sets included bag of words, and 2- and 3-term n-grams. Citation portions included titles, titles and abstracts, and full citations with metadata. Phase II with optimization involved a subset of the classifiers, as well as features extracted from full citations, and full citations with overweighted titles. We optimized features and classifier parameters by manually setting information gain thresholds outside of a process for iterative grid optimization with 10-fold cross-validations. We independently tested models on data reserved for that purpose and statistically compared classifier performance on 2 types of feature sets. We estimated the number of citations needed to screen by reviewers during a second pass through a reduced set of citations. Results In phase I, the evolutionary support vector machine returned the best recall for bag of words extracted from full citations; the best classifier with respect to overall performance was k-nearest neighbor. No classifier attained good enough recall for this task without optimization. In phase II, we boosted performance with optimization for evolutionary support vector machine and complement naïve Bayes classifiers. Generalization performance was better for the latter in the independent tests. For evolutionary support vector machine and complement naïve Bayes classifiers, the initial retrieval set was reduced by 46% and 35%, respectively. Conclusions Machine learning classifiers can help identify nonrandomized studies eligible for full-text screening by systematic reviewers. Optimization can markedly improve performance of classifiers. However, generalizability varies with the classifier. The number of citations to screen during a second independent pass through the citations can be substantially reduced.

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

“…Mean recall was 95% and above in [4,5,17,30,48,56,74,77,82] while it was below 95% in [9,49]. Precision on the other hand was over 10% in [4,5,17,30,48,49,82].…”

“…Precision on the other hand was over 10% in [4,5,17,30,48,49,82]. AUC was used in [12,18,48,55,60,82] and the result was over 0.5 in all the studies.…”

“…Some studies also experimented with multiple n-grams [4,6,12,14,16]. Figure 3 shows the techniques and the number of times each was used across all the studies.…”

Friedman and Wilcoxon Evaluations Comparing SVM, Bagging, Boosting, K-NN and Decision Tree Classifiers

“…Statistical tests for machine learning algorithms on several data sets using Wilcoxon signed ranks test and F detailed in [7] The raisins superiority for agriculture is graded by means of machine learning selecting the best features using feature selection based on correlation in [8]. Data from wir sensors for the job of physical movement recognition is taken for comparing the p AdaBoostM1 as the classifier of meta level with base level classifier C4.5 Graft in [9] classifier performance with optimization to categorize non-randomized readings and c biomedical quotations for text selection using organized reviews are studied in [10]. Cla Support vector machines, Conditional Random fields and Latent Dynamic conditional ran compared for user intention understanding in analysing web search engines was shown in [1 were analysed, sampled, selected and predicted for taxonomic soil class after investigating th power of data mining classifiers in [12].…”

“…Data from wireless kinematic sensors for the job of physical movement recognition is taken for comparing the performance of AdaBoostM1 as the classifier of meta level with base level classifier C4.5 Graft in [9]. Investigating classifier performance with optimization to categorize non-randomized readings and classification of biomedical quotations for text selection using organized reviews are studied in [10]. Classifiers namely Support vector machines, Conditional Random fields and Latent Dynamic conditional random fields are compared for user intention understanding in analysing web search engines was shown in [11].…”