Strengthening the foundation of educational psychology by integrating construct validation into open science reform

Flake, Jessica Kay

doi:10.1080/00461520.2021.1898962

Cited by 46 publications

(38 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rather, measures show greater or lesser evidence of validity for a certain use on a certain population within a certain context (Gehlbach, 2015). So if validity can only properly be assessed after data describing the participants and their context are collected and analyzed, how will open-science practices like preregistration and registered reports that require a priori assumptions work (Flake, 2021/ By no means are these the only challenges for educational psychologists in adopting and adapting open-science practices. Other challenges arise around specific practices.…”

Section: Additional Challenges and Future Tensionsmentioning

confidence: 99%

From old school to open science: The implications of new research norms for educational psychology and beyond

Gehlbach

Robinson

2021

Educational Psychologist

View full text Add to dashboard Cite

Recently, scholars have noted how several "old-school" practices-a host of long-standing scientific norms-in combination, sometimes compromise the credibility of research. In response, other scholarly fields have developed several "open-science" norms and practices to address these credibility issues. Against this backdrop, this special issue explores the extent to which and how these norms should be adopted and adapted for educational psychology and education more broadly. Our introductory article contextualizes the special issue's goals by overviewing the historical context that led to open science norms (particularly in medicine and psychology); providing a conceptual map to illustrate the interrelationships between various old-school as well as open-science practices; and then describing educational psychologists' opportunity to benefit from and contribute to the translation of these norms to novel research contexts. We conclude by previewing the articles in the special issue. Adopting and adapting open scienceCognitively, we knew that the mere act of preregistering studies was not enough to ensure the level of rigor we sought in our work. But the painful experiential learning gained from these replication attempts proved to be a much

show abstract

Section: Additional Challenges and Future Tensionsmentioning

confidence: 99%

From old school to open science: The implications of new research norms for educational psychology and beyond

Gehlbach

Robinson

2021

Educational Psychologist

View full text Add to dashboard Cite

show abstract

“…However, ad hoc measures are commonly used without any evidence of their validity or without any reasoned justification as to why they are appropriate (Flake & Fried, 2020). Accordingly, they may appear face valid, but can be problematic because they do not pass through a rigorous process of test construction and psychometric analysis to formally establish their validity (Flake, 2021; Flake et al, 2017). Given that study conclusions depend on the valid measurement of the studied constructs, the use of unvalidated and unjustified ad hoc measures has even been described as a “questionable measurement practice” (Flake & Fried, 2020, p. 458).…”

Section: Measuring Goal Dimensionsmentioning

confidence: 99%

“…Given that study conclusions depend on the valid measurement of the studied constructs, the use of unvalidated and unjustified ad hoc measures has even been described as a “questionable measurement practice” (Flake & Fried, 2020, p. 458). Questionable measurement practices have been claimed to be common in many subdisciplines of psychology, including social, personality, and educational psychology (Flake, 2021; Flake et al, 2017). Our impression that there seems to be little agreement on how to measure goal dimensions and the claim that questionable measurement practices are common in psychology gave us reasons to review whether such practices are present in the measurement of goals.…”

Section: Measuring Goal Dimensionsmentioning

confidence: 99%

The measurement of goal dimensions: A critical review.

Kiendl¹,

Hennecke²

2022

Motivation Science

View full text Add to dashboard Cite

Goals represent a central construct in the analysis of human motivation. How people appraise their own goals on various quantitative dimensions, like commitment or difficulty, predicts important outcomes like goal attainment and psychological well-being. Unfortunately, however, there appears to be little agreement in the literature on how to best measure goal dimensions. To get an overview of current measurement practices, we conducted a systematic literature review that yielded 693 publications including 1,166 different terms for goal dimensions and 3,244 reported items. Up to 79 different items were found to indicate single goal dimensions and 80% of scales consisted of one item, which mostly asked for the term of the goal dimension name. We further extracted 358 different item facets, that is, different meaning-bearing terms (e.g., “engrossed” or “deeply involved”). Of these item facets, 28% were found only once in individual items, and 52% were used to measure more than one goal dimension. We found that for 42% of items, sources were not reported, or items were created ad hoc. Hence, we recommend being cautious of jingle fallacies when comparing similar goal dimensions across publications and being cautious of jangle fallacies when comparing different goal dimensions. We advocate considering construct validity and content validity more thoroughly in the development of goal dimension items and enhancing transparency in reporting of study items. Furthermore, we advocate factor analysis methodology to narrow down the abundance of goal dimension terms and developing parsimonious and comprehensive theories of goal dimensions.

show abstract

“…With direct relevance for open science practices, however, more focused attention on the "art" of replication seems warranted. For example, how does one identify studies worthy of replication (see, Makel et al, 2012), and to what extent should measurement issues inform these decisions (Flake, 2021/this issue)? In addition, how does one ascertain the reasons for failures to replicate: was the original study or the replication flawed; how many replications are necessary before a finding is considered proven or disproven?…”

Section: Strengths and Limitations Of Replication As A Tool For Open Science Reformmentioning

confidence: 99%

Open science reforms: Strengths, challenges, and future directions

Wentzel

2021

Educational Psychologist

View full text Add to dashboard Cite

In this article, I comment on the potential benefits and limitations of open science reforms for improving the transparency and accountability of research, and enhancing the credibility of research findings within communities of policy and practice. Specifically, I discuss the role of replication and reproducibility of research in promoting better quality studies, the identification of generalizable principles, and relevance for practitioners and policymakers. Second, I suggest that greater attention to theory might contribute to the impact of open science practices, and discuss ways in which theory has implications for sampling, measurement, and research design. Ambiguities concerning the aims of preregistration and registered reports also are highlighted. In conclusion, I discuss structural roadblocks to open science reform and reflect on the relevance of these reforms for educational psychology.

show abstract

Strengthening the foundation of educational psychology by integrating construct validation into open science reform

Abstract: as a part of a special issue titled, "Educational psychology in the open science era."This preprint has been peer reviewed, but not copy edited by the journal and may differ from the final published version. The DOI of the final published version is

Cited by 46 publications

References 54 publications

From old school to open science: The implications of new research norms for educational psychology and beyond

From old school to open science: The implications of new research norms for educational psychology and beyond

The measurement of goal dimensions: A critical review.

Open science reforms: Strengths, challenges, and future directions

Contact Info

Product

Resources

About