Risk of bias and open science practices insystematic reviews of educational effectiveness:A meta-review

Abstract

BackgroundObjectives: In order to produce the most reliable syntheses of the effectiveness of educational interventions, systematic reviews need to adhere to rigorous methodological standards. This meta-review investigated risk of bias and presence of open science practices like data sharing and reproducibility of the review procedure, in recently published reviews in education. MethodsEligibility criteria: We included all systematic reviews of educational interventions, instructions and methods for all K-12 student populations in any school form with experimental or quasi-experimental designs (an active manipulation of the intervention) with comparisons and where the outcome variables were academic performance of any kind. Information sources: We searched the database Education Resources Information Center (ERIC) through the years 2019-2021. In parallel we hand-searched four major educational review journals for systematic reviews: Educational research review (Elsevier), Educational review (Taylor & Francis), Review of Education (Wiley) and Review of educational research (AERA).Risk of bias: Systematic reviews were formally assessed by the tool ROBIS - Risk of bias in systematic reviews.Synthesis of results: Systematic reviews were assessed with the risk of bias tool ROBIS and whether the studies had pre-registered protocols, shared primary research data, and if a third party could reproduce search strings and details of where exactly primary research data was extracted from. ResultsIncluded studies: From a total of 88 studies that matched our PICOS, 18 systematic reviews passed a shortened ROBIS assessment and were subjected to the full ROBIS assessment.Synthesis of results: 10 educational systematic reviews were judged as low risk of bias from a total of 88 systematic reviews. This means that approximately 11 percent of the systematic reviews were of low risk of bias. The rest were classified as high risk of bias during the shortened ROBIS or assessed as high risk or unclear risk of bias following the full ROBIS assessment. Of the 10 low risk of bias reviews, 6 had detailed their search sufficiently enough for a third party to reproduce, 3 reviews shared the data from primary studies, however none had specified how and from where exactly data from primary studies were extracted.DiscussionInterpretation: The study shows that at least a small part of systematic reviews in education are of high quality, as shown by being assessed as low risk of bias, but most systematic reviews in this area are of lower quality. There are still improvements in this field to be expected as even the low risk of bias reviews are not consistent regarding pre-registered protocols, data sharing, reproducibility of primary research data and reproducible search strings. Funding: Swedish Research Council, 2020-03430Registration: https://doi.org/10.17605/OSF.IO/WXFDJ