Analyzing Student Response Processes to Evaluate Success on a Technology-Based Problem-Solving Task

Han, Yuting; Wilson, Mark

doi:10.1080/08957347.2022.2034821

Cited by 5 publications

(2 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, it might not always correspond to the total time spent on an item, such as when students moved back and forth across the items (OECD, 2019). If more information (e.g., at the item level) is provided and utilized, response sequences within/across test units can be investigated to better understand the behavioral patterns (e.g., Han & Wilson, 2022;He et al, 2021;He et al, 2022). It may also be possible to examine the specific number of thresholds and upper limits for effective behavioral patterns with respect to the shared mental models and the extent of their similarities and dissimilarities.…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

“…Indeed, analyzing test-takers' process data makes it possible to understand the characteristics of performance in depth, for instance, how 15-year-old students interacted in problem spaces, such as incorrect responses despite overall effective strategies or correct responses that relied on guessing (He et al, 2022;Teig et al, 2020). However, such studies are still in the embryonic stage and have mostly revolved around the relatively small-to medium-scale assessments with the human-to-human approach (i.e., H-H approach), which entails naturalistic collaboration with people (e.g., Andrews-Todd et al, 2018;Dowell et al, 2018;Han & Wilson, 2022;Hao & Mislevy, 2019;Hu & Chen, 2022). Little research has been carried out on the process data from large-scale assessments that have used the H-A approach, such as the one employed in PISA 2015.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Behavioral patterns in collaborative problem solving: a latent profile analysis based on response times and actions in PISA 2015

Han,

Krieger,

Borgonovi

et al. 2023

Large-scale Assess Educ

View full text Add to dashboard Cite

Process data are becoming more and more popular in education research. In the field of computer-based assessments of collaborative problem solving (ColPS), process data have been used to identify students’ test-taking strategies while working on the assessment, and such data can be used to complement data collected on accuracy and overall performance. Such information can be used to understand, for example, whether students are able to use a range of styles and strategies to solve different problems, given evidence that such cognitive flexibility may be important in labor markets and societies. In addition, process information might help researchers better identify the determinants of poor performance and interventions that can help students succeed. However, this line of research, particularly research that uses these data to profile students, is still in its infancy and has mostly been centered on small- to medium-scale collaboration settings between people (i.e., the human-to-human approach). There are only a few studies involving large-scale assessments of ColPS between a respondent and computer agents (i.e., the human-to-agent approach), where problem spaces are more standardized and fewer biases and confounds exist. In this study, we investigated students’ ColPS behavioral patterns using latent profile analyses (LPA) based on two types of process data (i.e., response times and the number of actions) collected from the Program for International Student Assessment (PISA) 2015 ColPS assessment, a large-scale international assessment of the human-to-agent approach. Analyses were conducted on test-takers who: (a) were administered the assessment in English and (b) were assigned the Xandar unit at the beginning of the test. The total sample size was N = 2,520. Analyses revealed two profiles (i.e., Profile 1 [95%] vs. Profile 2 [5%]) showing different behavioral characteristics across the four parts of the assessment unit. Significant differences were also found in overall performance between the profiles.

show abstract

Section: Limitations and Future Directionsmentioning

confidence: 99%

mentioning

confidence: 99%

Behavioral patterns in collaborative problem solving: a latent profile analysis based on response times and actions in PISA 2015

Han,

Krieger,

Borgonovi

et al. 2023

Large-scale Assess Educ

View full text Add to dashboard Cite

show abstract

An Exploratory Approach to Predicting Performance in an Online Electronics Collaborative Problem‐Solving Task

Steinberg,

Forsyth,

Andrews‐Todd

2024

ETS Research Report Series

View full text Add to dashboard Cite

In a study of 370 postsecondary students in electronics, engineering, and other science classes, we investigated collaborative problem‐solving (CPS) skills that best predict performance at individual levels in an online electronics environment. The results showed that while monitoring was a consistent predictor across levels, other skills such as executing, sharing information, planning, and maintaining communication each predicted individual performance at one or more levels of the task. The availability of background data on students' content classes and associated content knowledge to analyze the model results can help identify possible cues for instructors across domains to help students improve specific CPS skills to achieve high performance in activities conducted in collaborative learning environments.

show abstract

Polytomous Effectiveness Indicators in Complex Problem-Solving Tasks and Their Applications in Developing Measurement Model

Wang,

Liu

2024

Psychometrika

View full text Add to dashboard Cite

Recent years have witnessed the emergence of measurement models for analyzing action sequences in computer-based problem-solving interactive tasks. The cutting-edge psychometrics process models require pre-specification of the effectiveness of state transitions often simplifying them into dichotomous indicators. However, the dichotomous effectiveness becomes impractical when dealing with complex tasks that involve multiple optimal paths and numerous state transitions. Building on the concept of problem-solving, we introduce polytomous indicators to assess the effectiveness of problem states ds\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{s}$$\end{document} and state-to-state transitions Δds→s′\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm {\Delta }d}_{\mathrm {s\rightarrow s'}}$$\end{document}. The three-step evaluation method for these two types of indicators is proposed and illustrated across two real problem-solving tasks. We further present a novel psychometrics process model, the sequential response model with polytomous effectiveness indicators (SRM-PEI), which is tailored to encompass a broader range of problem-solving tasks. Monte Carlo simulations indicated that SRM-PEI performed well in the estimation of latent ability and transition tendency parameters across different conditions. Empirical studies conducted on two real tasks supported the better fit of SRM-PEI over previous models such as SRM and SRMM, providing rational and interpretable estimates of latent abilities and transition tendencies through effectiveness indicators. The paper concludes by outlining potential avenues for the further application and enhancement of polytomous effectiveness indicators and SRM-PEI.

show abstract

Analyzing Student Response Processes to Evaluate Success on a Technology-Based Problem-Solving Task

Cited by 5 publications

References 30 publications

Behavioral patterns in collaborative problem solving: a latent profile analysis based on response times and actions in PISA 2015

Behavioral patterns in collaborative problem solving: a latent profile analysis based on response times and actions in PISA 2015

An Exploratory Approach to Predicting Performance in an Online Electronics Collaborative Problem‐Solving Task

Polytomous Effectiveness Indicators in Complex Problem-Solving Tasks and Their Applications in Developing Measurement Model

Contact Info

Product

Resources

About