Active learning-based Systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders

Abstract

Systematic reviews and meta-analyses are top of the bill in research. However, the screening phase requires an enormous effort in reading and labeling thousands of papers identified via systematic search. Active learning-aided systematic reviewing offers a solution by combining machine learning algorithms with user input to reduce screening load. This study explores the performance of these algorithms and different ways to apply them. This study is divided into four studies evaluating and improving this active learning pipeline. First, the performance and stability of the active learning pipeline were assessed via simulations and re-analysis of the outcome. Secondly, a convolutional neural network was developed to improve upon available machine learning algorithms. Thirdly, the performance of different algorithm combinations was tested and compared. Finally, algorithm-switching models were built for increased performance. The study concludes with proposals for improving active learning-aided systematic reviews based on combinations of the four studies. It was found that switching models can outperform the currently used models.

AI-aided Systematic Review to Create a Database with Potentially Relevant Papers on Depression, Anxiety, and Addiction

“…Firstly, the results of simulation studies were applied to new data. For example, which model to use in the first screening phase was decided based on a simulation study performed on a different, smaller, dataset about depression (Teijema et al, 2022).…”

“…Therefore, for the second screening phase, we first used the labeling decisions of the first round to optimize the hyperparameters per topic to receive the optimal hyperparameters for a 17-layer convolutional neural network (CNN) (Teijema, 2021) in combination with a doc2vec feature extractor. This model appeared to have better performance than the default deep learning models available in ASReview as was concluded in a simulation study conducted on similar data (Teijema et al, 2022). Using the optimized hyperparameters, we trained the 17-layer CNN model.…”

“…In the first screening phase, we used logistic regression as the classifier and TF-IDF as the feature extractor. The settings of the active learning model were based on a simulation study performed by Teijema et al (2022) on the depression dataset of Brouwer et al (2019).…”

“…Simulation studies show that machine-learning-based prioritization with active learning enables us to find relevant studies much faster than traditional screening methods; it can save up to 95% of screening time ( Van de Schoot, De Bruin, Schram, Zahedi, De Boer, Weijdema, Kramer, Huijts, Ferdinands, Harkema, Harkema, Willemsen, Ma, Fang, Sybren, et al, 2021). For example, a simulation study was performed on labeled data retrieved from a previously described meta-analysis (Brouwer et al, 2019) on the prospective evidence for leading psychological theories of depressive relapse (Teijema et al, 2022). The total number of papers found in the search was 50.936, of which 63 were included in the final meta-analysis (excluding the papers found with snowballing).…”

Reproducibility and Data storage Checklist for Active Learning-Aided Systematic Reviews

“…Lastly, we could take an even broader view and argue that the set of included records is also allowed to vary as long as the conclusion coming out of the post-processing stage is still the same. Reproducing a systematic review in this way gives confidence that missing a specific record is not essential for the final conclusions; see for an example [33].…”