Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks

Abstract

Interactive tasks designed to elicit real-life problem-solving behavior are rapidly becoming more widely used in educational assessment. Incorrect responses to such tasks can occur for a variety of different reasons such as low proficiency levels, low metacognitive strategies, or motivational issues. We demonstrate how behavioral patterns associated with incorrect responses can, in part, be understood, supporting insights into the different sources of failure on a task. To this end, we make use of sequence mining techniques that leverage the information contained in time-stamped action sequences commonly logged in assessments with interactive tasks for (a) investigating what distinguishes incorrect behavioral patterns from correct ones and (b) identifying subgroups of examinees with similar incorrect behavioral patterns. Analyzing a task from the Programme for the International Assessment of Adult Competencies 2012 assessment, we find incorrect behavioral patterns to be more heterogeneous than correct ones. We identify multiple subgroups of incorrect behavioral patterns, which point toward different levels of effort and lack of different subskills needed for solving the task. Albeit focusing on a single task, meaningful patterns of major differences in how examinees approach a given task that generalize across multiple tasks are uncovered. Implications for the construction and analysis of interactive tasks as well as the design of interventions for complex problem-solving skills are derived.

Examining Humans’ Problem-Solving Styles in Technology-Rich Environments Using Log File Data

“…However, the average response time was far longer than five seconds, which was used as a constant threshold for the minimum amount of time needed to validly respond to a task (e.g., Goldhammer et al 2016 ; Wise and Kong 2005 ). In this respect, the Shirking style is different from disengaged test-taking behavior, though being disengaged is common in low-stakes assessments, such as the PIAAC 2012 ( Goldhammer et al 2016 ; Ulitzsch et al 2021 ). Since various factors (e.g., cognition and personality) may impact how people respond to technology-based problems ( Feist and Barron 2003 ), future studies should collect more data to explore what factors are associated with the presence of the three problem-solving styles in TRE.…”

“…Considering feasibility reasons, TRE in the present study are limited to settings involving the most common digital technologies ( Nygren et al 2019 ): computers (e.g., spreadsheet) and Internet-based services (e.g., web browser). To boost the use of digital technologies, a bulk of research has investigated factors that might affect humans’ problem-solving performance in TRE (e.g., Liao et al 2019 ; Millar et al 2020 ; Nygren et al 2019 ; Ulitzsch et al 2021 ). Among those findings, problem-solving style was regarded as one of the most prominent factors (e.g., Koć-Januchta et al 2020 ; Lewis and Smith 2008 ; Treffinger et al 2008 ).…”

“…According to Selby et al ( 2004, p. 222 ), problem-solving styles are “consistent individual differences in the ways people prefer to plan and carry out generating and focusing activities, in order to gain clarity, produce ideas, and prepare for action”. This broadly accepted definition indicates that problem-solving style derives from one’s distinguishable behavioral pattern (e.g., He et al 2021 ; Ulitzsch et al 2021 ). In this regard, problem-solving styles in TRE reflect individuals’ dispositions regarding how they are inclined to interact with surrounding technology environments.…”

Identifying dynamic shifts to careless and insufficient effort behavior in questionnaire responses; A novel approach and experimental validation

“…Jin et al (2018) used such an approach where they specified a class representing CNR behavior by modeling a random response pattern with equal probabilities for each Likert response category. Similar approaches use person-fit indices (e.g., Terzi, 2017) or external information such as response times to improve classification (Ulitzsch, He, & Pohl, 2022;Molenaar & de Boeck, 2018). In the latter approach, response times from online surveys are used to enrich class predictions.…”

Editorial: Process Data in Educational and Psychological Measurement

“…These sorts of data stored in log files, referred to as process data in this book, provide information beyond response data that typically show response accuracy only. This additional information holds promise to help us understand the strategies that underlie test performance and identify key actions that lead to success or failure of answering an item (e.g., Han et al, 2019 ; Liao et al ; Stadler et al, 2019 ; He et al, 2021 ; Ulitzsch et al, 2021a ; Xiao et al, 2021 ).…”