The Campbell Collaboration’s systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards

Abstract

Background Many published reviews do not meet the widely accepted PRISMA standards for systematic reviews and meta-analysis. Campbell Collaboration and Cochrane reviews are expected to meet even more rigorous standards, but their adherence to these standards is uneven. For example, a newly updated Campbell systematic review of school-based anti-bullying interventions does not appear to meet many of the Campbell Collaboration’s mandatory methodological standards. Issues In this commentary, we document methodological problems in the Campbell Collaboration's new school-based anti-bullying interventions review, including (1) unexplained deviations from the protocol; (2) inadequate documentation of search strategies; (3) inconsistent reports on the number of included studies; (4) undocumented risk of bias ratings; (5) assessments of selective outcome reporting bias that are not transparent, not replicable, and appear to systematically underestimate risk of bias; (6) unreliable assessments of risk of publication bias; (7) use of a composite scale that conflates distinct risks of bias; and (8) failure to consider issues related to the strength of the evidence and risks of bias in interpreting results and drawing conclusions. Readers who are unaware of these problems may place more confidence in this review than is warranted. Campbell Collaboration editors declined to publish our comments and declined to issue a public statement of concern about this review. Conclusions Systematic reviews are expected to use transparent methods and follow relevant methodological standards. Readers should be concerned when these expectations are not met, because transparency and rigor enhance the trustworthiness of results and conclusions. In the tradition of Donald T. Campbell, there is need for more public debate about the methods and conclusions of systematic reviews, and greater clarity regarding applications of (and adherence to) published standards for systematic reviews.

Tutkimus koulukiusaamista koskevasta asiantuntijapuheesta

“…Sittemmin edellä mainitun meta-analyysin tekijät ovat tehneet uuden metaanalyysin kiusaamisinterventioiden tehokkuudesta (Gaffney ym., 2021). Sen tilastollinen luotettavuus on tullut kyseenalaistetuksi muun muassa siksi, että kriteerit sille, miten aineisto on valittu mukaan, eivät ole läpinäkyvät, ja sen vuoksi tutkimuksessa mukana olevien kiusaamisen vastaisten interventioiden toimivuudesta voi tulla liian positiivinen tulos ja analyysin luotettavuus voi vaikuttaa suuremmalta kuin se on (Littell & Gorman, 2022). Ajankohtaisessa tutkimuksessa on myös nostettu esiin kysymys siitä, kuinka hyvin ylipäätään kvantitatiivisen tutkimuksen avulla voidaan tutkia monimutkaisia sosiaalisia prosesseja tai esimerkiksi lasten ja nuorten omia näkemyksiä kiusaamisesta (esim.…”

PROTOCOL: Assessment of outcome reporting bias in studies included in Campbell systematic reviews

“…ORB ratings appear to be perfunctory in some reviews, indicating that there is room for improvement in methods of assessing ORB. For example, a prominent Campbell review (Gaffney et al, 2021) assessed ORB as "low risk" in most included studies but provided no support for these ratings; independent attempts to replicate these ratings (using the review's stated criteria) with 41 studies showed very low (7%) agreement (κ = 0.003; Littell & Gorman, 2022).…”