Exploring latent states of problem‐solving competence using hidden Markov model on process data

Abstract

The response process of problem‐solving items contains rich information about respondents' behaviours and cognitive process in the digital tasks, while the information extraction is a big challenge. The aim of the study is to use a data‐driven approach to explore the latent states and state transitions underlying problem‐solving process to reflect test‐takers' behavioural patterns, and to investigate how these states and state transitions could be associated with test‐takers' performance. We employed the Hidden Markov Modelling approach to identify test takers' hidden states during the problem‐solving process and compared the frequency of states and/or state transitions between different performance groups. We conducted comparable studies in two problem‐solving items with a focus on the US sample that was collected in PIAAC 2012, and examined the correlation between those frequencies from two items. Latent states and transitions between them underlying the problem‐solving process were identified and found significantly different by performance groups. The groups with correct responses in both items were found more engaged in tasks and more often to use efficient tools to solve problems, while the group with incorrect responses was found more likely to use shorter action sequences and exhibit hesitative behaviours. Consistent behavioural patterns were identified across items. This study demonstrates the value of data‐driven based HMM approach to better understand respondents' behavioural patterns and cognitive transmissions underneath the observable action sequences in complex problem‐solving tasks.

Unsupervised Anomaly Detection in Sequential Process Data

“…The information obtained from log files has been used for addressing a wide range of research questions, such as identifying problem-solving strategies (Stadler et al, 2019), exploring different patterns of behaviors (Hahnel et al, 2022; Zhu et al, 2019), and measuring test-takers’ ability, engagement, and motivation levels (Nagy et al, 2022; Xiao et al, 2021). However, only a few studies have utilized process data for anomaly detection (e.g., Gorgun & Bulut, 2022; Liao et al, 2021).…”

Subtask analysis of process data through a predictive model

“…HMMs have been used in many fields such as natural language processing and bioinformatics for segmentation (Yamron et al, 1997; Yoon, 2009). Recently, they have been used for exploring the latent states in response processes (Xiao et al, 2021). In our context, response processes and subtask sequences are treated as the observed sequences and the hidden state sequences, respectively.…”

“…A latent variable model for response processes has been proposed by Chen (2020). Xiao et al (2021) explored the latent states in problem-solving processes using hidden Markov models.…”

20 years of interactive tasks in large‐scale assessments: Process data as a way towards sustainable change?

“…This comprehensive switch from paper‐pencil assessments to technology‐based assessments has led to some rather technical improvements such as identifying early guessing (e.g., Kong et al, 2007) or improving standardization of assessment and scoring (e.g., Goldhammer et al, 2020). At the same time, process data on student interaction with items have been shown to carry value for obtaining, reporting, and interpreting additional results on student skills in international comparisons (e.g., Reis Costa et al, 2021; Xiao et al, 2021). Process data was used to relate behaviour to cognitive processes (Greiff et al, 2016), to validate score interpretations (Kane & Mislevy, 2017), and led to a better theoretical understanding of the construct under investigation (Goldhammer et al, 2017; Goldhammer & Zehner, 2017).…”