Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments

Abstract

Objective To compare the effectiveness of two search methods in identifying studies that used the Control Preferences Scale (CPS), a healthcare decision-making instrument commonly used in clinical settings. Study Design & Setting We searched the literature using two methods: 1) keyword searching using variations of “control preferences scale” and 2) cited reference searching using two seminal CPS publications. We searched three bibliographic databases [PubMed, Scopus, Web of Science (WOS)] and one full-text database (Google Scholar). We report precision and sensitivity as measures of effectiveness. Results Keyword searches in bibliographic databases yielded high average precision (90%), but low average sensitivity (16%). PubMed was the most precise, followed closely by Scopus and WOS. The Google Scholar keyword search had low precision (54%) but provided the highest sensitivity (70%). Cited reference searches in all databases yielded moderate sensitivity (45–54%), but precision ranged from 35–75% with Scopus being the most precise. Conclusion Cited reference searches were more sensitive than keyword searches, making it a more comprehensive strategy to identify all studies that use a particular instrument. Keyword searches provide a quick way of finding some but not all relevant articles. Goals, time and resources should dictate the combination of which methods and databases are used.

Effect of exercise training on C reactive protein: a systematic review and meta-analysis of randomised and non-randomised controlled trials

“…However traditional, a keyword search of multiple databases may only identify 75% of relevant records, with no single database able to identify all relevant publications 72 73. Including a manual search of citations in addition to a systematic search of bibliographic databases offers a more comprehensive search strategy, and may mitigate potential methodological concerns related to the sensitivity of electronic keyword searches 74. As such, the interdisciplinary nature of physical activity, exercise and biomedical research requires the search of multiple databases using a range of relevant terms and synonyms, coupled with a manual search of relevant records in order to perform an exhaustive search of the literature.…”

Frailty assessment instruments: Systematic characterization of the uses and contexts of highly-cited instruments

“…Though PubMed is an outstanding search tool for keyword search (Linder et al, 2015), we selected the Web of Science (WoS) for our citation searches based on (a) its emphasis on top tier journals and high quality publications (http://wokinfo.com/essays/journal-selection-process/) and (b) its longer period of citation analysis coverage when compared to other databases (Falagas et al, 2008). Additionally PubMed includes less journals in its database than WoS (Falagas et al, 2008).…”

Conduct and reporting of citation searching in Cochrane systematic reviews: A cross‐sectional study

“…9 Studies that compare the effectiveness of citation searching with searching bibliographic databases show that citation searching is particularly effective at retrieving studies for systematic reviews where core concepts are difficult to capture using keywords, eg, where core concepts are described inconsistently due to systemic reporting deficiencies, or due to historical development of terminology in a subject area or research methodology. 9,[11][12][13][14][15] Iterative citation searching using studies identified by citation searching, or citation snowballing, might be useful for systematic reviews of hard-to-find studies, such as those included in qualitative evidence synthesis. 16 In these types of review, citation searching can be considered a complementary or even primary search method rather than as supplementary to searching bibliographic databases.…”

“…To date, reviews on the conduct and reporting of search methods in systematic reviews have reported findings on citation searching relatively briefly-typically, the prevalence of the search method. 10,21 There are several published case studies of citation searching [11][12][13][14]18 and reviews of such studies, 3,9 but their focus is the effectiveness of the search method compared with other search methods (usually in a single case study) rather than a cross-sectional analysis of conduct and reporting. We chose to examine Cochrane reviews in particular because they are a more consistently high standard of systematic review than other types of systematic review.…”

“…The aim of citation searching is to identify studies missed by text‐based searches in the title, abstract, or controlled vocabulary fields of bibliographic records . Studies that compare the effectiveness of citation searching with searching bibliographic databases show that citation searching is particularly effective at retrieving studies for systematic reviews where core concepts are difficult to capture using keywords, eg, where core concepts are described inconsistently due to systemic reporting deficiencies, or due to historical development of terminology in a subject area or research methodology . Iterative citation searching using studies identified by citation searching, or citation snowballing , might be useful for systematic reviews of hard‐to‐find studies, such as those included in qualitative evidence synthesis .…”