Open science reforms: Strengths, challenges, and future directions

Abstract

In this article, I comment on the potential benefits and limitations of open science reforms for improving the transparency and accountability of research, and enhancing the credibility of research findings within communities of policy and practice. Specifically, I discuss the role of replication and reproducibility of research in promoting better quality studies, the identification of generalizable principles, and relevance for practitioners and policymakers. Second, I suggest that greater attention to theory might contribute to the impact of open science practices, and discuss ways in which theory has implications for sampling, measurement, and research design. Ambiguities concerning the aims of preregistration and registered reports also are highlighted. In conclusion, I discuss structural roadblocks to open science reform and reflect on the relevance of these reforms for educational psychology.

What Can Educational Psychology Learn From, and Contribute to, Theory Development Scholarship?

“…Reform movements have included calls for a more open science with greater methodological rigor (Fleming et al, 2021;Gehlbach & Robinson, 2021;Munafò et al, 2017) and a commitment to anti-racist and inclusive scholarship and practices (APA, 2021;Buchanan et al, 2021;Zusho & Kumar, 2018). Recently, these critically important calls have been echoed by a somewhat less prominent, but nonetheless complementary, call for reform in the ways psychologists engage in theory development (Eronen & Bringmann, 2021;Oberauer & Lewandowsky, 2019;Plaut, 2010;Wentzel, 2021).…”

“…Theory, and how it is developed, directly relate to psychology's methodological and anti-racist goals because theory guides who and what psychologists study, the questions psychologists ask, how psychologists interpret findings, and what implications psychologists suggest based on those findings (Eronen & Bringmann, 2021;Szollsi & Donkin, 2021). Yet, theory development has been relatively underscrutinized compared to the field's focus on methods (Fielder, 2017;Wentzel, 2021). Scholars who have looked deeply at psychology's use of theory have lamented the paucity of clear descriptions of theory development (Eronen & Bringmann, 2021), the scarcity of clear connections between theory and the empirical work published in psychology journals, (McPhetres et al, 2021), the difficulty in evaluating the adequacy of theories (Gervais, 2021), the oftenunchecked proliferation of multiple seemingly similar theories about the same phenomena (Eronen & Romeihn, 2020;Mischel, 2008), and the inequitable prominence of WEIRD (i.e., Western, educated, industrialized, rich, and democratic;Henrich et al, 2010) and able-ist (Emery & Anderman, 2020) perspectives in psychology scholarship.…”

“…The need for tripartite reform (i.e., methods, anti-racism, theory development) in psychology extends to its many subdisciplines (Eronen & Bringmann, 2021), with educational psychology being no exception (DeCuir-Gunby & Schutz, 2014;Makel et al, 2021;Wentzel, 2021). There is emerging evidence of questionable research practices in education and educational psychology (Gehlbach & Robinson, 2021) as well as continued evidence the field must reckon with past failures to address epistemic injustice (Kidd et al, 2017) based on race (Lopez, 2022;Usher, 2018) and ability (Emery & Anderman, 2020), among other factors.…”

“…For example, after publishing a critique of theory development in psychology writ large, Gigerenzer (2010) reported receiving this response from a professor: "No field within psychology suffers more from the theoretical malaise than mine, educational psychology" (p. 734). More recently, Wentzel (2021) argued "an important challenge for research reform is to achieve greater parsimony across theoretical frameworks" (p. 166) and that efforts at methodological reform in educational psychology "also will require changes in how we train the next generation of researchers to incorporate theory into sampling, measurement, design, and data analysis decisions" (p. 169).…”

“…Without effective integration, a field can experience unchecked and likely unsustainable growth in the number of theories, amusingly characterized by Mischel (2008): "psychologists treat other peoples' theories like toothbrushes -no self-respecting person wants to use anyone else 's" (n.d.). Excessive growth in the number of theories can hinder educational psychology's ability to generate reliable findings with practical implications, jeopardizing its reputation with practitioners, policy-makers, and the public (Gehlbach & Robinson, 2021;Wentzel, 2021;Willingham, 2017).…”