Null and Void? Errors in Meta-analysis on Perceptual Disfluency and Recommendations to Improve Meta-analytical Reproducibility

Abstract

In the 2018 meta-analysis of Educational Psychology Review entitled “Null effects of perceptual disfluency on learning outcomes in a text-based educational context” by Xie, Zhou, and Liu, we identify some errors and inconsistencies in both the methodological approach and the reported results regarding coding and effect sizes. While from a technical point of view the meta-analysis aligns with current meta-analytical guidelines (e.g., PRISMA) and conforms to general meta-analytical requirements (e.g., considering publication bias), it exemplifies certain insufficient practices in the creation and review of meta-analysis. We criticize the lack of transparency and negligence of open-science practices in the generation and reporting of results, which complicate evaluation of the meta-analytical reproducibility, especially given the flexibility in subjective choices regarding the analytical approach and the flexibility in creating the database. Here we present a framework applicable to pre- and post-publication review on improving the Methods Reproducibility of meta-analysis. Based on considerations of the transparency and openness (TOP)-guidlines (Nosek et al. Science 348: 1422–1425, 2015), the Reproducibility Enhancement Principles (REP; Stodden et al. Science 354:1240–1241, 2016), and recommendations by Lakens et al. (BMC Psychology 4: Article 24, 2016), we outline Computational Reproducibility (Level 1), Computational Verification (Level 2), Analysis Reproducibility (Level 3), and Outcome Reproducibility (Level 4). Applying reproducibility checks to TRANSFER performance as the chosen outcome variable, we found Xie’s and colleagues’ results to be (rather) robust. Yet, regarding RECALL performance and the moderator analysis, the identified problems raise doubts about the credibility of the reported results.

Is this going to be on the test? Test expectancy moderates the disfluency effect with sans forgetica.

“…Thus, Sans Forgetica (and perceptual disfluency manipulations in general) may not effectively enhance memory in ecologically valid settings. Although a recent meta-analysis (Xie et al, 2018;Weissgerber et al, 2021) claimed the disfluency effect was null and void, what is clear from the current findings is that the impact of perceptual disfluency manipulations such as Sans Forgetica is not straightforward. Researchers should heed the call to examine further the conditions under which perceptual disfluency is and not desirable for learning.…”

“…Although perceptual disfluency has the potential to be valuable (and easy to implement), a recent meta-analysis has called into question whether perceptual disfluency is desirable for learning (Xie et al, 2018; cf. Weissgerber et al, 2021). The current research investigates under what conditions disfluency is and is not beneficial for learning using Sans Forgetica as a proxy for perceptual disfluency.…”

Foreign experience in evaluating the efficiency of scientific and pedagogical research

“…Складнiсть вiдтворюваностi може бути зумовлена не тiльки специфiкою окремих дослiдницьких проєктiв, але i людськими факторами, зокрема, випадковими або i зумисними хибними твердженнями про iстиннiсть результатiв дослiджень. Авторами Weissgerber S. C., Brunmair M., Rummer R. [41] запропоновано поетапну логiку забезпечення вiдтворюваностi психолого-педагогiчних дослiджень, що передбачає перевiрку числових результатiв при незалежному вiдтвореннi, вивчення коду статистичних опрацювань, аналiзi отриманих даних i пропонованих висновкiв.…”

“…Науковi розвiдки аналiзу зарубiжного досвiду щодо оцiнювання результативностi науково-педагогiчних дослiджень надали можливiсть виокремити такi актуальнi напрями: визначення критерiїв наукового дослiдження ( [21], [31], [34], [41] та iн. ), використання наукометричних баз даних (НБД) у процесi оцiнювання дослiджень ( [5], [7], [22], [35] та iн.…”

Sans Forgetica font may help memory for words but not for numbers, nor does it help analytical thinking

“…They found no effect of perceptual disfluency on recall (d = SF VS. ARIAL: WORDS, NUMBERS, ANALYTIC THINKING 4 -0.01) or transfer (d=0.03) and no evidence of moderators (participant characteristics, learning material, experimental design) for an effect. However, their meta-analysis is under some recent scrutiny -Weissgerber, Brunmair, & Rummer (2021) find combining covariate-corrected results and uncorrected results in moderator analyses problematic and suggest obtaining raw values from primary studies. Another meta-analysis examined the effect of disfluent font on people's abilities to solve Cognitive Reflection Test (CRT) problems, where the most intuitive solution to problems is incorrect and deeper processing is needed to reach the correct solution.…”