Performance decline in low-stakes educational assessments: different mixture modeling approaches

List, M.; Robitzsch, Alexander; Lüdtke, Oliver; Köller, Olaf; Nagy, Gabriel

doi:10.1186/s40536-017-0049-3

Cited by 16 publications

(18 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In line with the literature, we assumed in the present research that RGB is a type of behavior that, similar to other phenomena in the testing context (e.g., item position effects, performance decline; e.g., Hartig and Buchholz, 2012; Debeer et al, 2014; Jin and Wang, 2014; List et al, 2017; Weirich et al, 2017; Wise and Gao, 2017; Nagy et al, 2018b), has a high probability of being maintained (at a student level) over the course of a test session, once it has begun. This means that once individuals engage in RGB, they have a high probability of showing this behavior in the subsequent items of the test.…”

Section: Introductionsupporting

confidence: 68%

“…This suggests that physical and/or mental fatigue plays a role in reduced test-taking effort (Lindner et al, 2018), which may also explain why the most important predictor of RGB is the elapsed testing time (see e.g., Wise et al, 2009). There is compelling evidence across studies that items presented in later positions in a test are typically solved with lower accuracy (item position effect; e.g., List et al, 2017; Weirich et al, 2017; Nagy et al, 2018a), less motivational effort (e.g., Barry and Finney, 2016; Penk and Richter, 2017) and are substantially more prone to RGB (e.g., Wise et al, 2009; Setzer et al, 2013;Goldhammer et al, 2016).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Onset of Rapid-Guessing Behavior Over the Course of Testing Time: A Matter of Motivation and Cognitive Resources

2019

Self Cite

View full text Add to dashboard Cite

Digital tests make it possible to identify student effort by means of response times, specifically, unrealistically fast responses that are defined as rapid-guessing behavior (RGB). In this study, we used latent class and growth curve models to examine (1) how student characteristics (i.e., gender, school type, general cognitive abilities, and working-memory capacity) are related to the onset point of RGB and its development over the course of a test session (i.e., item positions). Further, we examined (2) the extent to which repeated ratings of task enjoyment (i.e., intercept and slope parameters) are related to the onset and the development of RGB over the course of the test. For this purpose, we analyzed data from N = 401 students from fifth and sixth grades in Germany ( n = 247 academic track; n = 154 non-academic track). All participants solved 36 science items under low-stakes conditions and rated their current task enjoyment after each science item, constituting a micro-longitudinal design that allowed students' motivational state to be tracked over the entire test session. In addition, they worked on tests that assessed their general cognitive abilities and working-memory capacity. The results show that students' gender was not significantly related to RGB but that students' school type (which is known to be closely related to academic abilities in the German school system), general cognitive abilities, and their working-memory capacity were significant predictors of an early RGB onset and a stronger RGB increase across testing time. Students' initial rating of task enjoyment was associated with RGB, but only a decline in students' task enjoyment was predictive of earlier RGB onset. Overall, non-academic-school attendance was the most powerful predictor of RGB, together with students' working-memory capacity. The present findings add to the concern that there is an unfortunate relation between students' test-effort investment and their academic and general cognitive abilities. This challenges basic assumptions about motivation-filtering procedures and may threaten a valid interpretation of results from large-scale testing programs that rely on school-type comparisons.

show abstract

Section: Introductionsupporting

confidence: 68%

Section: Introductionmentioning

confidence: 99%

The Onset of Rapid-Guessing Behavior Over the Course of Testing Time: A Matter of Motivation and Cognitive Resources

2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Rapid guessing occurs when test takers run out of time at the end of a test and respond to the last items fast with little reasoning. Similar to previous publications—see List, Robitzsch, Lüdtke, Köller, and Nagy (2017) for an overview—we assumed a sudden change from regular responding to rapid guessing in the last items of the test. Data were simulated as follows.…”

Section: Simulation Studymentioning

confidence: 96%

Robust Estimation of Ability and Mental Speed Employing the Hierarchical Model for Responses and Response Times

Ranger

Kuhn

Wolgast

2020

J Educational Measurement

View full text Add to dashboard Cite

Van der Linden's hierarchical model for responses and response times can be used in order to infer the ability and mental speed of test takers from their responses and response times in an educational test. A standard approach for this is maximum likelihood estimation. In real‐world applications, the data of some test takers might be partly irregular, resulting from rapid guessing or item preknowledge. The maximum likelihood estimator is not robust against contamination with irregular data. In this article, we propose a robust estimator of ability and mental speed. The estimator consists of two steps. In the first step, the mental speed is estimated with the estimator of Gervini and Yohai that ignores outlying response times. In the second step, the ability is estimated with an M‐estimator that down weights unusual responses given at unusual response times. This is achieved by combining the hard‐rejection weights of Gervini and Yohai with the M‐estimator suggested by Croux and Haesbroeck for the logistic regression model. The proposed estimator is consistent, almost as efficient as the maximum likelihood estimator in uncontaminated data and robust in contaminated data. The performance of the estimator is analyzed in a simulation study and an empirical example.

show abstract

“…Further findings suggested item position effects on test performance (Weirich et al, 2016;Nagy et al, 2018Nagy et al, , 2019Rose et al, 2019;Liu and Hau, 2020). A problem found was decreased test performance over the course of taking a computer-assisted achievement test (List et al, 2017). That raised the question if motivation similarly decreased over the course of taking a computer-assisted achievement test.…”

Section: Introductionmentioning

confidence: 99%

Test-Taking Motivation in Education Students: Task Battery Order Affected Within-Test-Taker Effort and Importance

Wolgast¹,

Schmidt

Ranger

2020

Front. Psychol.

View full text Add to dashboard Cite

Different types of tasks exist, including tasks for research purposes or exams assessing knowledge. According to expectation-value theory, tests are related to different levels of effort and importance within a test taker. Test-taking effort and importance in students decreased over the course of high-stakes tests or low-stakes-tests in research on test-taking motivation. However, whether test-order changes affect effort, importance, and response processes of education students have seldomly been experimentally examined. We aimed to examine changes in effort and importance resulting from variations in test battery order and their relations to response processes. We employed an experimental design assessing N = 320 education students’ test-taking effort and importance three times as well as their performance on cognitive ability tasks and a mock exam. Further relevant covariates were assessed once such as expectancies, test anxiety, and concentration. We randomly varied the order of the cognitive ability test and mock exam. The assumption of intraindividual changes in education students’ effort and importance over the course of test taking was tested by one latent growth curve that separated data for each condition. In contrast to previous studies, responses and test response times were included in diffusion models for examining education students’ response processes within the test-taking context. The results indicated intraindividual changes in education students’ effort or importance depending on test order but similar mock-exam response processes. In particular effort did not decrease, when the cognitive ability test came first and the mock exam subsequently but significantly decreased, when the mock exam came first and the cognitive ability test subsequently. Diffusion modeling suggested differences in response processes (separation boundaries and estimated latent trait) on cognitive ability tasks suggesting higher motivational levels when the cognitive ability test came first than vice versa. The response processes on the mock exam tasks did not relate to condition.

show abstract

Performance decline in low-stakes educational assessments: different mixture modeling approaches

Cited by 16 publications

References 38 publications

The Onset of Rapid-Guessing Behavior Over the Course of Testing Time: A Matter of Motivation and Cognitive Resources

The Onset of Rapid-Guessing Behavior Over the Course of Testing Time: A Matter of Motivation and Cognitive Resources

Robust Estimation of Ability and Mental Speed Employing the Hierarchical Model for Responses and Response Times

Test-Taking Motivation in Education Students: Task Battery Order Affected Within-Test-Taker Effort and Importance

Contact Info

Product

Resources

About