Combining Clickstream Analyses and Graph-Modeled Data Clustering for Identifying Common Response Processes

Ulitzsch, Esther; He, Qiwei; Ulitzsch, Vincent; Molter, Hendrik; Nichterlein, André; Niedermeier, Rolf; Pohl, Steffi

doi:10.1007/s11336-020-09743-0

Cited by 36 publications

(36 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This suggests that partially and fully correct action sequences are more similar to each other and thus can be collapsed into one group. Following Ulitzsch et al (2021a), actions that are not essential to successfully solve the task were recoded by aggregate-level categories (e.g., “responding to an email,” “seeking help,” “keystrokes,” “creating new folder,” “using the toolbar,” “opening folders”). In addition, email and folder identifiers were dropped.…”

Section: Data and Materialsmentioning

confidence: 99%

“…For disentangling incorrect behavioral patterns, we drew on an exploratory two-step approach that was proposed in Ulitzsch et al (2021a). This hybrid method combines data mining techniques originally developed for the analysis of clickstream data (see Banerjee & Ghosh, 2001) with graph-modeled data clustering for identifying common and dominant behavioral patterns.…”

Section: Analysesmentioning

confidence: 99%

See 1 more Smart Citation

Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks

Ulitzsch

Pohl

2021

Journal of Educational and Behavioral Statistics

Self Cite

View full text Add to dashboard Cite

Interactive tasks designed to elicit real-life problem-solving behavior are rapidly becoming more widely used in educational assessment. Incorrect responses to such tasks can occur for a variety of different reasons such as low proficiency levels, low metacognitive strategies, or motivational issues. We demonstrate how behavioral patterns associated with incorrect responses can, in part, be understood, supporting insights into the different sources of failure on a task. To this end, we make use of sequence mining techniques that leverage the information contained in time-stamped action sequences commonly logged in assessments with interactive tasks for (a) investigating what distinguishes incorrect behavioral patterns from correct ones and (b) identifying subgroups of examinees with similar incorrect behavioral patterns. Analyzing a task from the Programme for the International Assessment of Adult Competencies 2012 assessment, we find incorrect behavioral patterns to be more heterogeneous than correct ones. We identify multiple subgroups of incorrect behavioral patterns, which point toward different levels of effort and lack of different subskills needed for solving the task. Albeit focusing on a single task, meaningful patterns of major differences in how examinees approach a given task that generalize across multiple tasks are uncovered. Implications for the construction and analysis of interactive tasks as well as the design of interventions for complex problem-solving skills are derived.

show abstract

Section: Data and Materialsmentioning

confidence: 99%

Section: Analysesmentioning

confidence: 99%

Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks

Ulitzsch

Pohl

2021

Journal of Educational and Behavioral Statistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…For instance, Sahin and Colvin (2020) suggested to combine information on the time spent on task with information on the number of performed actions to classify responses as rapid, disengaged responses. Ulitzsch, He, and Pohl (2021) proposed using sequence mining techniques by combining action sequences and time intervals between actions to understand the incorrect behavioural patterns on interactive tasks. It would be interesting to leverage these observable variables with the latent states and transitions from the HMM application in the future studies.…”

Section: Discussionmentioning

confidence: 99%

“…Even though item developers and content experts could pre‐define optimal strategies that are the most efficient ways to solve the tasks, the strategies may not be exclusively recorded in advance (He et al, 2019). Also, there are always unpredictable behaviour patterns in the incorrect response group (Ulitzsch et al, 2021; Ulitzsch, He, & Pohl, 2021). Second, process data are generally noisy, highly variable, and hard to perceive (Tang et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

Exploring latent states of problem‐solving competence using hidden Markov model on process data

Xiao

Veldkamp

et al. 2021

Computer Assisted Learning

Self Cite

View full text Add to dashboard Cite

The response process of problem‐solving items contains rich information about respondents' behaviours and cognitive process in the digital tasks, while the information extraction is a big challenge. The aim of the study is to use a data‐driven approach to explore the latent states and state transitions underlying problem‐solving process to reflect test‐takers' behavioural patterns, and to investigate how these states and state transitions could be associated with test‐takers' performance. We employed the Hidden Markov Modelling approach to identify test takers' hidden states during the problem‐solving process and compared the frequency of states and/or state transitions between different performance groups. We conducted comparable studies in two problem‐solving items with a focus on the US sample that was collected in PIAAC 2012, and examined the correlation between those frequencies from two items. Latent states and transitions between them underlying the problem‐solving process were identified and found significantly different by performance groups. The groups with correct responses in both items were found more engaged in tasks and more often to use efficient tools to solve problems, while the group with incorrect responses was found more likely to use shorter action sequences and exhibit hesitative behaviours. Consistent behavioural patterns were identified across items. This study demonstrates the value of data‐driven based HMM approach to better understand respondents' behavioural patterns and cognitive transmissions underneath the observable action sequences in complex problem‐solving tasks.

show abstract

“…For instance, recently researchers proposed different models for response time and the joint modeling of responses and response time (e.g., Bolsinova and Molenaar ; Costa et al ; Wang et al ). In addition, other process data such as the path collected based on eye-tracking devices (e.g., Zhu and Feng, 2015 ; Maddox et al, 2018 ; Man and Harring, 2021 ), action sequences in problem-solving tasks (e.g., Chen et al ; Tang et al, 2020 ; He et al, 2021 ; Ulitzsch et al, 2021b ), and processes in collaborative problem solving (e.g., Graesser et al, 2018 ; Andrews-Todd and Kerr, 2019 ; De Boeck and Scalise, 2019 ), are also worthy of exploration and integration with product data for assessment purposes.…”

mentioning

confidence: 99%

Editorial: Process Data in Educational and Psychological Measurement

Jiao

Veldkamp

2021

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

The increasing use of computer-based testing and learning environments is leading to a significant reform on the traditional form of measurement, with tremendous extra available data collected during the process of learning and assessment (Bennett et al., 2007(Bennett et al., , 2010. It means that we can learn and describe the respondents' performances not only by their responses, but also their responding processes, in addition to the response accuracy in the traditional tests (Ercikan and Pellegrino, 2017).The recent advances in computer technology enhance the convenient collection of process data in computer-based assessment. One such example is time-stamped action data in an innovative item which allow for the interaction between a respondent and the item. When a respondent attempts an interactive item, his/her actions are recorded, in the form of an ordered sequence of multi-type, time-stamped events. These sorts of data stored in log files, referred to as process data in this book, provide information beyond response data that typically show response accuracy only. This additional information holds promise to help us understand the strategies that underlie test performance and identify key actions that lead to success or failure of answering an item (e.g.,

show abstract

Combining Clickstream Analyses and Graph-Modeled Data Clustering for Identifying Common Response Processes

Cited by 36 publications

References 53 publications

Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks

Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks

Exploring latent states of problem‐solving competence using hidden Markov model on process data

Editorial: Process Data in Educational and Psychological Measurement

Contact Info

Product

Resources

About