An alternative to the hand searching gold standard: validating methodological search filters using relative recall

Abstract

BackgroundSearch filters or hedges play an important role in evidence-based medicine but their development depends on the availability of a "gold standard" – a reference standard against which to establish the performance of the filter. We demonstrate the feasibility of using relative recall of included studies from multiple systematic reviews to validate methodological search filters as an alternative to validation against a gold standard formed through hand searching.MethodsWe identified 105 Cochrane reviews that used the Highly Sensitive Search Strategy (HSSS), included randomized or quasi-randomized controlled trials, and reported their included studies. We measured the ability of two published and one novel variant of the HSSS to retrieve the MEDLINE-index studies included in these reviews.ResultsThe systematic reviews were comprehensive in their searches. 72% of included primary studies were indexed in MEDLINE. Relative recall of the three strategies ranged from .98 to .91 across all reviews and more comprehensive strategies showed higher recall.ConclusionAn approach using relative recall instead of a hand searching gold standard proved feasible and produced recall figures that were congruent with previously published figures for the HSSS. This technique would permit validation of a methodological filter using a collection of approximately 100 studies of the chosen design drawn from the included studies of multiple systematic reviews that used comprehensive search strategies.

Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study

“…hand-searching database records or sets of publications (usually journals) to identify all of the records that meet a set of explicit criteria 69,133 2. using the RR technique to create a reference set based on the results of a systematic review. 52 …”

“…Using the RR technique to create a reference set of publications, based on the results of a systematic review, has been described in detail by Sampson et al 52 RR has been used to develop reference sets for testing search filter performance by a number of researchers (reviews A and B). 2,48,49 The studies included in a systematic review (or other research project), in which extensive searching using sensitive search strategies and other approaches to study identification have been employed, are taken as a quasi-reference set.…”

“…Using multiple reviews has been suggested to increase the robustness of the reference set. 52 The RR reference set will have been created at a specific point in time; the same subject search run subsequently will find more results and it is difficult to recreate the status of a database at a specific point in time. Methods to remove later studies, to approximate to the state of the database at the time of the original searches, may be used but should be documented.…”

“…Methods exist to test search filter performance and to build the performance picture, including reviews of search filter performance. 48,49,[52][53][54] There is no formal guidance, however, on the best methods for testing filter performance, on which performance measures are valued by searchers and on which measures should ideally be reported to assist searchers in choosing between filters. The performance picture for filters across different disciplines, questions and databases is therefore largely unknown.…”

El filtro académico brasileño: aplicación a la literatura científica sobre actividad física

“…12 This demonstrates the need to create geographical or academic search fi lters that ensure effi cient access to this scientifi c literature. 15,19 Therefore, the objective of this study was to develop a search equation to retrieve the academic scientifi c work pertaining to the theme of physical activity.…”