The contribution of databases to the results of systematic reviews: a cross-sectional study

Abstract

BackgroundOne of the best sources for high quality information about healthcare interventions is a systematic review. A well-conducted systematic review includes a comprehensive literature search. There is limited empiric evidence to guide the extent of searching, in particular the number of electronic databases that should be searched. We conducted a cross-sectional quantitative analysis to examine the potential impact of selective database searching on results of meta-analyses.MethodsOur sample included systematic reviews (SRs) with at least one meta-analysis from three Cochrane Review Groups: Acute Respiratory Infections (ARI), Infectious Diseases (ID), Developmental Psychosocial and Learning Problems (DPLP) (n = 129). Outcomes included: 1) proportion of relevant studies indexed in each of 10 databases; and 2) changes in results and statistical significance of primary meta-analysis for studies identified in Medline only and in Medline plus each of the other databases.ResultsDue to variation across topics, we present results by group (ARI n = 57, ID n = 38, DPLP n = 34). For ARI, identification of relevant studies was highest for Medline (85 %) and Embase (80 %). Restricting meta-analyses to trials that appeared in Medline + Embase yielded fewest changes in statistical significance: 53/55 meta-analyses showed no change. Point estimates changed in 12 cases; in 7 the change was less than 20 %. For ID, yield was highest for Medline (92 %), Embase (81 %), and BIOSIS (67 %). Restricting meta-analyses to trials that appeared in Medline + BIOSIS yielded fewest changes with 1 meta-analysis changing in statistical significance. Point estimates changed in 8 of 31 meta-analyses; change less than 20 % in all cases. For DPLP, identification of relevant studies was highest for Medline (75 %) and Embase (62 %). Restricting meta-analyses to trials that appeared in Medline + PsycINFO resulted in only one change in significance. Point estimates changed for 13 of 33 meta-analyses; less than 20 % in 9 cases.ConclusionsMajority of relevant studies can be found within a limited number of databases. Results of meta-analyses based on the majority of studies did not differ in most cases. There were very few cases of changes in statistical significance. Effect estimates changed in a minority of meta-analyses but in most the change was small. Results did not change in a systematic manner (i.e., regularly over- or underestimating treatment effects), suggesting that selective searching may not introduce bias in terms of effect estimates.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-016-0232-1) contains supplementary material, which is available to authorized users.

Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews

“…Moreover, the results of meta-analyses based on studies contained in fewer databases did not differ, in the majority of cases, from the results of meta-analyses that included all identified studies [23]. However, we noted that the choice of database, and likewise decisions around searching for and including non-English studies, unpublished studies, and dissertations are likely topic dependent.…”

Supplementary search methods were more effective and offered better value than bibliographic database searching: A case study from public health and environmental enhancement

“…Searching end‐to‐end of this methodological process seeks to address the risk of publication bias, because even those studies that are more difficult to identify are still sought, although in reality the time spent searching, using each individual search method, is often different and decreases after the primary method is undertaken. Hartling et al explore the possibility of prioritising which databases to search in systematic reviews, but we believe this study is the first to prioritise and allocate search methods, in particular, supplementary search methods, in a review.…”

Protocol for a Rapid Evidence Review of Traditional and Complementary Medicine for People with Diabetes Receiving Palliative or End-of-Life Care

“…In a crosssectional analyses of 129 SRs, an exhaustive literature search of up to ten databases was not shown to significantly alter the estimated effect sizes compared to SRs of only two databases (EMBASE and Medline). 71 Furthermore the inclusion of all possible trials of any quality may potentially increase the risk of bias due to the effects of low quality studies skewing the results. 70 …”