Techniques for identifying cross-disciplinary and ‘hard-to-detect’ evidence for systematic review

Abstract

Driven by necessity in our own complex review, we developed alternative systematic ways of identifying relevant evidence where the key concepts are generally not focal to the primary studies' aims and are found across multiple disciplines-that is, hard-to-detect evidence. Specifically, we sought to identify evidence on community engagement in public health interventions that aim to reduce health inequalities. Our initial search strategy used text mining to identify synonyms for the concept 'community engagement'. We conducted a systematic search for reviews on public health interventions, supplemented by searches of trials databases. We then used information in the reviews' evidence tables to gather more information about the included studies than was evident in the primary studies' own titles or abstracts. We identified 319 primary studies cited in reviews after full-text screening. In this paper, we retrospectively reflect on the challenges and benefits of the approach taken. We estimate that more than a quarter of the studies that were identified would have been missed by typical searching and screening methods. This identification strategy was highly effective and could be useful for reviews of broad research questions, or where the key concepts are unlikely to be the main focus of primary research.

Text mining for search term development in systematic reviewing: A discussion of some methods and challenges

“…O'Mara‐Eves et al observed that a small challenge is deciding on the threshold below which terms identified by Termine would not be considered, and they used a threshold C‐value of 5 when analysing the full text of 5 papers. Their rationale was that “it was the common value below which mined terms seemed to lose relevance across the five papers” (p. 53).…”

“…The data included in sample are also important to consider. In the examples here, citations and abstracts were used; however, O'Mara‐Eves et al used the full‐text of 5 papers that were seminal within the area of their search focus. In some cases, the use of full‐texts may not always be possible (for example, limitations in the software, or a lack of known relevant studies) or it might be too inefficient to make the process worthwhile, given that retrieving and then the processing the full‐text documents may add considerable time.…”

“…Although presented here as distinct applications, these applications could be used in combination and iteratively during search term development, or perhaps to analyse subcomponents of a search. They could also potentially be used on a set of screened records, either as quality assurance or as part of an integrated searching and screening system …”

Systematic Reviews: A Social Work Perspective

“…For example, where the topic for review cuts across disciplines, different terminologies may be used (O'Mara-Eves et al, 2014). Similarly, Littell (2006) reported that search strategies that worked for her UK colleagues needed adaptation in North America.…”

Citation analysis as a literature search method for systematic reviews

“…As a result, a growing number of librarians and informatics practitioners have suggested the use of analytics to facilitate this process. Many of these suggestions involve the use of text mining and topic modeling to automatically generate search queries to identify relevant documents (Aphinyanaphongs, Tsamardinos, Statnikov, Hardin, & Aliferis, 2005;Hausner, Waffenschmidt, Kaiser, & Simon, 2012;O'Mara-Eves et al, 2014) or to categorize the retrieved documents to facilitate the literature screening process (Adeva, Atxa, Carrillo, & Zengotitabengoa, 2014;Bekhuis & Demner-Fushman, 2012;Frunza, Inkpen, Matwin, Klement, & O'Blenis, 2011;Shemilt et al, 2014;Wallace, Trikalinos, Lau, Brodley, & Schmid, 2010). For overviews of text mining applications in systematic reviews, see Thomas, McNaught, and Ananiadou (2011); Lefebvre, Glanville, Wieland, Coles, and Weightman (2013); and O'Mara-Eves, Thomas, McNaught, Miwa, and Ananiadou (2015).…”