Effects of Motivation on the Accuracy and Speed of Responding in Tests: The Speed-Accuracy Tradeoff Revisited

Ranger, Jochen; Kuhn, Jörg‐Tobias; Pohl, Steffi

doi:10.1080/15366367.2020.1750934

Cited by 13 publications

(9 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interestingly, we also found an interaction between participant ability and item difficulty: whereas the worst-performing participants maintained the same work rate irrespective of item difficulty, the best-performing participants spent more time deliberating as items became more challenging. These results are consistent with a persistence interpretation of ability, wherein good task performance in part reflects a willingness to invest time in the solution process and poor task performance reflects a tendency to give up sooner (Ranger, Kuhn, & Pohl, 2021). As such, the MaRs-IB may be suitable not only for measuring matrix reasoning ability but also mental effort costs (Kool & Botvinick, 2018), opportunity costs (Payne, Bettman, & Luce, 1996), or other motivational factors (Duckworth et al, 2011) related to people's tendency to exert effort or give up.…”

Section: Discussionsupporting

confidence: 81%

An item response theory analysis of the Matrix Reasoning Item Bank (MaRs-IB)

Zorowitz

Chierchia²,

Blakemore³

et al. 2022

Preprint

View full text Add to dashboard Cite

Matrix reasoning tasks are among the most widely used measures of cognitive ability in the behavioral sciences, but the paucity of matrix reasoning tests in the public domain complicates their use. Here we present an extensive investigation and psychometric validation of the matrix reasoning item bank (MaRs-IB), an open-access set of matrix reasoning items. In a first study, we calibrate the psychometric functioning of the items in the MaRs-IB in a large sample adults participants (N=1501). Using additive multilevel item structure models, we establish that the MaRs-IB has many desirable psychometric properties: its items span a wide range of difficulty, possess medium- to-large levels of discrimination, and exhibit robust associations between item complexity and difficulty. However, we also find that item clones are not always psychometrically equivalent and cannot assumed to be exchangeable. In a second study, we demonstrate how researchers can use the item parameters we estimated to design new MaRs-IB tests using optimal item assembly. With these methods, we design three parallel short-form measures that exhibited good score reliability and convergent validity in a second sample of adults (N=300). We hope that the materials, results, and tutorials made available here will encourage researchers to use the MaRs-IB in their studies.

show abstract

Section: Discussionsupporting

confidence: 81%

An item response theory analysis of the Matrix Reasoning Item Bank (MaRs-IB)

Zorowitz

Chierchia²,

Blakemore³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We aim to study, in a variety of data, whether the general intuition behind the SAT holds. Conceptually, this study builds on work suggesting that additional time spent on a response does not always increase its accuracy (Bolsinova & Molenaar, 2018; Goldhammer et al, 2014; Ranger et al, 2021). For example, Chen et al (2018) discuss a curvilinear relationship between RT and accuracy: Increases in time spent on an item were associated with increases in accuracy, but only up to a certain point.…”

Section: Introductionmentioning

confidence: 98%

Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings

Domingue¹,

Kanopka²,

Stenhaug³

et al. 2022

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

The speed–accuracy trade-off (SAT) suggests that time constraints reduce response accuracy. Its relevance in observational settings—where response time (RT) may not be constrained but respondent speed may still vary—is unclear. Using 29 data sets containing data from cognitive tasks, we use a flexible method for identification of the SAT (which we test in extensive simulation studies) to probe whether the SAT holds. We find inconsistent relationships between time and accuracy; marginal increases in time use for an individual do not necessarily predict increases in accuracy. Additionally, the speed–accuracy relationship may depend on the underlying difficulty of the interaction. We also consider the analysis of items and individuals; of particular interest is the observation that respondents who exhibit more within-person variation in response speed are typically of lower ability. We further find that RT is typically a weak predictor of response accuracy. Our findings document a range of empirical phenomena that should inform future modeling of RTs collected in observational settings.

show abstract

“…The use of response time data is one of the most established applications of response process data in assessment, with a degree of formalization about its methods (Lee & Jia, 2014; Li et al, 2017; Wise, 2019) and related discussion of validation (Goldhammer et al, 2021; Li et al, 2017; Reis Costa et al, 2021; Wise, 2017, 2019). The digital transition has witnessed extensive application, such as measures to support inferences about rapid guessing behaviors and disengagement in large-scale assessments (Ercikan et al, 2020; Goldhammer et al, 2017; Kroehne et al, 2020; Lundgren & Eklöf, 2020; Soland et al, 2018; Wise, 2017, 2019, 2020a, 2020b; Wise & Gao, 2017, Wise & Kong, 2005, Wise et al, 2019), and to explore the relationship between response times, accuracy and ability (e.g., Ivanova et al, 2020; Michaelides et al, 2020; Ranger et al, 2021; Reis Costa et al, 2021).…”

Section: Item Response Times and Disabilitymentioning

confidence: 99%

Process and Product in Computer-Based Assessments

Zumbo

Maddox

Care

2023

European Journal of Psychological Assessment

View full text Add to dashboard Cite

There is no consensus among assessment researchers about many of the central problems of response process data, including what is it and what is it comprised of. The Standards for Educational and Psychological Testing ( American Educational Research Association et al., 2014 ) locate process data within their five sources of validity evidence. However, we rarely see a conceptualization of response processes; rather, the focus is on the techniques and methods of assembling response process indices or statistical models. The method often overrides clear definitions, and, as a field, we may therefore conflate method and methodology – much like we have conflated validity and validation ( Zumbo, 2007 ). In this paper, we aim to clear the conceptual ground to explore the scope of a holistic framework for the validation of process and product. We review prominent conceptualizations of response processes and their sources and explore some fundamental questions: Should we make a theoretical and practical distinction between response processes and response data? To what extent do the uses of process data reflect the principles of deliberate, educational, and psychological measurement? To answer these questions, we consider the case of item response times and the potential for variation associated with disability and neurodiversity.

show abstract

Effects of Motivation on the Accuracy and Speed of Responding in Tests: The Speed-Accuracy Tradeoff Revisited

Cited by 13 publications

References 70 publications

An item response theory analysis of the Matrix Reasoning Item Bank (MaRs-IB)

An item response theory analysis of the Matrix Reasoning Item Bank (MaRs-IB)

Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings

Process and Product in Computer-Based Assessments

Contact Info

Product

Resources

About