Things I Have Learned about Meta-Analysis Since 1990: Reducing Bias in Search of “The Big Picture” / Ce que j’ai appris sur la méta-analyse depuis 1990 : réduire les partis pris en quête d’une vue d’ensemble

Abstract

This paper examines sources of potential bias in systematic reviews and meta-analyses which can distort their findings, leading to problems with interpretation and application by practitioners and policymakers. It follows from an article that was published in the Canadian Journal of Communication in 1990, “Integrating Research into Instructional Practice: The Use and Abuse of Meta-analysis,” which introduced meta-analysis as a means for estimating population parameters and summarizing quantitative research around instructional research questions. This paper begins by examining two cases where multiple meta-analyses disagree. It then goes on to describe substantive and methodological aspects of meta-analysis where various kinds of bias can influence the outcomes and suggests measures that can be taken to avoid them. The intention is to improve the reliability and accuracy of reviews so that practitioners can trust the results and use them more effectively. Cet article examine les sources des partis pris potentiels dans les synthèses systématiques et les méta-analyses qui peuvent déformer les conclusions, ce qui peut causer des problèmes d’interprétation et d’application par les praticiens et les responsables des politiques. Il fait suite à un article publié dans le Canadian Journal of Communication en 1990, intitulé « Integrating Research into Instructional Practice: The Use and Abuse of Meta-analysis », qui présentait la méta-analyse comme moyen d’estimer les paramètres relatifs à la population et de résumer la recherche quantitative sur les questions de recherche pédagogique. L’article commence avec l’examen de deux cas dans lesquels de nombreuses méta-analyses sont en désaccord. Il décrit ensuite les aspects substantifs et méthodologiques des méta-analyses dans lesquels divers types de partis pris peuvent influencer les résultats et suggère des mesures qui peuvent être adoptées pour éviter ces partis pris. L’intention est d’améliorer la fiabilité et l’exactitude des synthèses afin que les praticiens puissent compter sur les résultats et les utiliser plus efficacement.

Systematically Searching Empirical Literature in the Social Sciences: Results from Two Meta-Analyses Within the Domain of Education

“…Librarians have also been at the forefront of attempts to identify areas for improvement in systematic review standards [24,26] and to formulate best practices and guidelines [e.g., 5,27]. These standards, if respected, will lead to better quality and less publication bias [18,28].…”

A study of meta-analyses reporting quality in the large and expanding literature of educational technology

“…Failure to do either would result in a biased meta-analysis. This being said, it is important to note that bias is not a unilateral aspect of a given meta-analysis but an outcome of one or more errors resulting from inadequate processes employed in data collection, manipulation, analysis, interpretation and/or presentation of research findings (Bernard, 2014). We consider flaws in any of these processes to be a methodological reporting bias that could potentially lead to presenting an inaccurate picture of the past research.…”