Both Questionable and Open Research Practices are Prevalent in Education Research

Abstract

Discussions of how to improve research quality are predominant in a number of fields, including education. But how prevalent are the use of problematic practices and the improved practices meant to counter them? This baseline information will be a critical data source as education researchers seek to improve our research practices. In this preregistered study, we replicated and extended previous studies from other fields by asking education researchers about 10 questionable research practices and 5 open research practices. We asked them to estimate the prevalence of the practices in the field, self-report their own use of such practices, and estimate the appropriateness of these behaviors in education research. We made predictions under four umbrella categories: comparison to psychology, geographic location, career stage, and quantitative orientation. Broadly, our results suggest that both questionable and open research practices are part of the typical research practices of many educational researchers. Preregistration, code, and data can be found at https://osf.io/83mwk/.

Questionable and open research practices: attitudes and perceptions among quantitative communication researchers

“…While the point estimates of QRP prevalence vary, John et al's (2012) findings have been replicated among samples of Brazilian, German and Italian psychologists (Agnoli, Wicherts, Veldkamp, Albiero, & Cubelli, 2017;Fiedler & Schwarz, 2016;Rabelo et al, 2020) and among researchers in ecology, evolutionary biology, and education (Fraser et al, 2018;Makel et al, 2019) -see Table 4 for an overview.…”

“…Vermeulen et al (2015, p. 202) concluded that empirical papers in communication most likely suffer from "a high false positive rate," likely driven by QRPs, such as only publishing studies that "worked," and placing those that did not work in a file drawer. According to surveys in other social science disciplines, non-publication is fairly common (Fraser, Parker, Nakagawa, Barnett, & Fidler, 2018;John et al, 2012;Makel, Hodges, Cook, & Plucker, 2019). Franco et al (2014) report direct evidence for non-publication among experiments that were fielded in the Time-sharing Experiments for the Social Sciences -a project where peer-reviewed experiments are fielded in a population based sample -and compared the reported number of conditions in the publication to the actual number of conditions that were included in the survey.…”

“…Adding or dropping variables to reach statistically significant results is another QRP difficult to track. But one that researchers in other fields indicate to do relatively often (John et al, 2012;Makel et al, 2019). Again, this practice is defensible if disclosed in an online appendix, for instance, but problematic if not made transparent.…”

The Value of Preregistration for Psychological Science: A Conceptual Analysis

“…Selective re-porting makes it more likely that a prediction is supported by the data, and less likely that a prediction is proven wrong. Given a scientific reward system where successful predictions are deemed more valuable than unsuccessful predictions it is perhaps not surprising that researchers admit to selectively reporting results (Fiedler & Schwarz, 2015;Fraser, Parker, Nakagawa, Barnett, & Fidler, 2018;John, Loewenstein, & Prelec, 2012;Makel, Hodges, Cook, & Plucker, 2019), and selectively submit significant results for publication (Franco, Malhotra, & Simonovits, 2014;Greenwald, 1975). This behavior, which violates most code of conducts for research integrity, but is nevertheless commonplace, leads to a scientific literature that does not reflect reality.…”

Les pratiques de recherche ouvertes en psychologie

“…Ils prennent la peine de rajouter que le simple fait de suggérer sur la base de résultats déformés que ce type de conduite est fréquent, affaiblit non seulement les messages de prévention destinés à prévenir ces pratiques de recherche discutables, mais nuisent également gravement à l'image de la recherche scientifique et par extension la confiance en ses recommandations (Fiedler & Schwarz, 2016). Là encore, ces pratiques de recherche discutables sont loin d'être uniquement propres à la psychologie, elles concernent par exemple l'écologie et la biologie de l'évolution (Fraser, Parker, Nakagawa, Barnett, & Fidler, 2018), les recherches biomédicales (Gardner, Lidz, & Hartwig, 2005), les sciences économiques (List, Bailey, Euzent, & Martin, 2001), les sciences de l'éducation (Makel, Hodges, Cook, & Plucker, 2019), le management (O'Boyle et al, 2017) ou encore l'ingénierie civile (Swazey, Anderson, Lewis, & Louis, 1993), etc. Si ces pratiques de recherche discutables paraissent à première vue inoffensives, elles sont de plus en plus montrées du doigt en raison de leur écart avec les nouveaux standards de recherche qui aspirent à une plus grande rigueur scientifique.…”