Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

Abstract

Rigorous evidence identification is essential for systematic reviews and meta‐analyses (evidence syntheses) because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science. A novel, query‐based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall, and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analyzed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. We call for database owners to recognize the requirements of evidence synthesis and for academic journals to reassess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.

Moral Foundations' Methodological Foundations: A Systematic Analysis of Reliability in Research Using the Moral Foundations Questionnaire

“…We purposely chose to conduct our search on Google Scholar, which indexes a large number of sources beyond academic journals, and were able to include a relatively large number of studies (published and unpublished) in the content analysis. However, researchers have expressed concerns about using Google Scholar as principal search system for systematic reviews due to issues with precision, recall, and reproducibility (Gusenbauer & Haddaway, 2020). Further, given the many translations of the MFQ into other languages, it is conceivable that there are studies published in other languages which we might have missed with our English-language search term.…”

A comparative analysis reveals irreproducibility in searches of scientific literature

“…While the disadvantages of the inconsistencies in Google Scholar search results have been repeatedly illustrated[18,19], the similar behavior from Web of Science has only recently been reported[13] but in neither case was the variability estimated nor were the potential solutions discussed. Given the widespread use of Web of Science, neglecting this discrepancy can mislead scientists when drawing conclusions from their evidence synthesis, when the body of evidence was collected by Web of Science searches alone.…”

“…With the advent and rapid development of Internet-based databases and search engines, the role of narrative reviews is now being overtaken by new, quantitative methods of evidence synthesis[11,12]. A core requirement in these activities, repeatability, crucially depends on reliable databases[13]. Large scientific databases/search engines, such as PubMed, Web of Science and Scopus, are essential in this process.…”

“…Differences among the most commonly used scientific search engines and databases are well documented[13,15,16] but knowledge of the consistency within databases in relation to geographical location where the search is requested from (but see Gusenbauer and Haddaway[13]), software environment, or computer configuration remain surprisingly limited. Since the search histories of users may be stored in the browsers’ cache, and considered by the scientific search engines, repeated and identical searches may result in different outcomes.…”

Modeling and Management Big Data in Databases—A Systematic Literature Review

“…Moreover, according to a comprehensive study, which evaluated the quality of 28 scientific search systems, Google Scholar is inappropriate as principal search system, while ScienceDirect, Scopus and WoS are suitable to evidence synthesis in an SLR [24].…”