The influence of active learning model and prior knowledge choice on how long it takes to find hard-to-find relevant papers: Examining the variability of the time to discovery and the stability of its rank-orders

Abstract

Tools that employ screening prioritization through active learning (AL) have accelerated the screening process significantly by ranking an unordered set of records by their predicted relevance. Failing to find a relevant paper might alter the findings of a systematic review, which highlights the importance of identifying hard-to-find relevant papers. The time to discovery (TD) measures how many records are needed to be screened to find a relevant paper, making it a useful tool for examining such papers. The main aim of this project was to investigate how the choice of the model and prior knowledge influence the TD values of the hard-to-find relevant papers and their rank orders. A simulation study was conducted mimicking the screening process on a dataset containing titles, abstracts, and labels used for a systematic review. The results demonstrated that AL model choice, and mostly the choice of the feature extractor but not the choice of prior knowledge, significantly influenced the TD values and the rank order of the TD values of the hard-to-find relevant papers. Future research should examine the characteristics of hard-to-find relevant papers to discover why they might take a long time to be found.