Understanding students' performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files

Greiff, Samuel; Niepel, Christoph; Scherer, Ronny; Martin, Romain

doi:10.1016/j.chb.2016.02.095

Cited by 126 publications

(109 citation statements)

References 33 publications

Supporting

Mentioning

102

Contrasting

Unclassified

Order By: Relevance

“…The potential relation between completion time and reading comprehension has, however, been analysed in a linear way. In future, a curvilinear analysis method, as has been reported in other domains (e.g., Greiff, Niepel, Scherer, & Martin, 2016), might further confirm or disconfirm this perceived change. The other process measures no longer contributed to the prediction of school performance beyond the prediction offered by accuracy.…”

Section: Discussionmentioning

confidence: 55%

“…In future, a curvilinear analysis method, as has been reported in other domains (e.g.,Greiff, Niepel, Scherer, & Martin, 2016), might further confirm or disconfirm this perceived change. For the prediction of reading comprehension, the amount of help provided during training provided more prediction than static test measures, but trained (posttest) performance did not.…”

mentioning

confidence: 92%

See 1 more Smart Citation

Process assessment in dynamic testing using electronic tangibles

Veerbeek

Vogelaar

Verhaegh³

et al. 2018

Computer Assisted Learning

View full text Add to dashboard Cite

Task solving processes and changes in these processes have long been expected to provide valuable information about children's performance in school. This article used electronic tangibles (concrete materials that can be physically manipulated) and a dynamic testing format (pretest, training, and posttest) to investigate children's task solving processes and changes in these processes as a result of training. We also evaluated the value of process information for the prediction of school results. Participants were N = 253 children with a mean age of 7.8 years. Half of them received a graduated prompts training; the other half received repeated practice only. Three process measures were used: grouping behaviour, verbalized strategies, and completion time. Different measures showed different effects of training, with verbalized strategies showing the largest difference on the posttest between trained and untrained children. Although process measures were related to performance on our dynamic task and to math and reading performance in school, the amount of help provided during training provided the most predictive value to school results. We concluded that children's task solving processes provide valuable information, but the interpretation requires more research.

show abstract

Section: Discussionmentioning

confidence: 55%

mentioning

confidence: 92%

Process assessment in dynamic testing using electronic tangibles

Veerbeek

Vogelaar

Verhaegh³

et al. 2018

Computer Assisted Learning

View full text Add to dashboard Cite

show abstract

“…Even though a growing number of references claiming to deal with complex problems exist (e.g., Greiff and Wüstenberg, 2015; Greiff et al, 2016), it would be better to label the requirements within these tasks “dynamic problem solving,” as it has been done adequately in earlier work (Greiff et al, 2012). The dynamics behind on-off-switches (Thimbleby, 2007) are remarkable but not really complex.…”

Section: What Is Not Cps?mentioning

confidence: 99%

Complex Problem Solving: What It Is and What It Is Not

2017

View full text Add to dashboard Cite

Computer-simulated scenarios have been part of psychological research on problem solving for more than 40 years. The shift in emphasis from simple toy problems to complex, more real-life oriented problems has been accompanied by discussions about the best ways to assess the process of solving complex problems. Psychometric issues such as reliable assessments and addressing correlations with other instruments have been in the foreground of these discussions and have left the content validity of complex problem solving in the background. In this paper, we return the focus to content issues and address the important features that define complex problems.

show abstract

“…However, this effect differed in that it was highly negative for easy items among higher-performing test-takers, but not high enough for difficult items and lower-performing test-takers. In another study (Greiff, Niepel, Scherer & Martin, 2016) using students' RT, it was revealed that spending an extremely low or high level of time led to lower performance in complex problem-solving. Lee and Haberman (2016) used RT to investigate test-taking behaviors in an international language assessment and found that the behaviors and RTs of examinees from different countries did not generally follow a stable trend.…”

mentioning

confidence: 99%

The Effects of Log Data on Students’ Performance

Yavuz

2019

Eğitimde Ve Psikolojide Ölçme Ve Değerlendirme Dergisi

View full text Add to dashboard Cite

This study aimed to assess the relationships between response times (RTs), the number of actions taken to solve a given item, and student performance. In addition, the interaction between the students' information and communications technology (ICT) competency, reading literacy, and log data (time and number of actions) were examined in order to gain additional insights regarding the relations between student performance and log data. The sample consisted of 2 348 students who participated in the triennial international large-scale assessment of the Programme for International Student Assessment (PISA). For the current study, 18 items in the one cluster of the 91st booklet were chosen. To achieve the aim of the study, explanatory item response modeling (EIRM) framework based on generalized linear mixed modeling (GLMM) was used. The results of this study showed that students who spent more time on items and those that took more actions on items were more likely to answer the items correctly. However, this effect did not have variability across items and students. Moreover, the interaction only with reading and the number of actions was found to have a positive effect on the students' overall performance.

show abstract

Understanding students' performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files

Cited by 126 publications

References 33 publications

Process assessment in dynamic testing using electronic tangibles

Process assessment in dynamic testing using electronic tangibles

Complex Problem Solving: What It Is and What It Is Not

The Effects of Log Data on Students’ Performance

Contact Info

Product

Resources

About