Operationalizing and Detecting Disengagement Within Online Science Microworlds

Abstract

In recent years, there has been increased interest in engagement during learning. This is of particular interest in the science, technology, engineering, and mathematics domains, in which many students struggle and where the United States needs skilled workers. This article lays out some issues important for framing research on this topic and provides a review of some existing work with similar goals on engagement in science learning. Specifically, here we seek to help better concretize engagement, a fuzzy construct, by operationalizing and detecting (i.e., identifying using a computational method) disengaged behaviors that are antithetical to engagement. We, in turn, describe our real-time detector (i.e., machine learned model) of disengaged behavior and how it was developed. Last, we address our ongoing research on how our detector of disengaged behavior will be used to intervene in real time to better support students' science inquiry learning in Inq-ITS (Inquiry-Intelligent Tutoring

Translating the ICAP Theory of Cognitive Engagement Into Practice

“…Despite our speculations of what the underlying cognitive knowledge‐change processes may entail while students interact with the learning materials, our definition of engagement is pragmatically based on the absence or presence of overt, observable behaviors, as well as whether the products contain information that went beyond the instructional materials (rather than based on identifying the underlying thinking processes per se). Defining engagement in this way is practical because (a) such overt behaviors and products can be more clearly operationalized by a straightforward comparison with instructional materials, as opposed to relying on judging the covert cognitive processes that may underlie the overt behaviors, (b) such definitions are more concrete for teachers to rely upon when they design lesson plans, and (c) such definitions, based on overt behaviors, allow teachers to more easily detect in situ whether students are appropriately engaged, and finally (d), behaviors and products are malleable and more easily elicited (consistent with the current view that engagement is malleable; Gobert et al., ; Reschly & Christenson, ).…”

Advanced, Analytic, Automated (AAA) Measurement of Engagement During Learning

“…From a theoretical perspective, the assumption that engagement is a multi-componential construct necessitates multimodal measurement as different modalities optimally index specific components. In particular, eye gaze and central physiology are best suited for cognitive engagement (Berka et al, 2007; Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995; Marshall, 2005; Rayner, 1998), facial features and peripheral physiology for affective engagement (Ekman, 1984; Keltner & Ekman, 2000; Larsen et al, 2008; Matsumoto et al, 2008), and interaction features for behavioral engagement (Baker & Ocumpaugh, 2015; Baker & Rossi, 2013; Bulger, Mayer, Almeroth, & Blau, 2008; Gobert et al, 2015). Multimodal measures that operate across multiple timescales ranging from milliseconds (physiological signals), milliseconds to seconds (bodily responses), and seconds to minutes (interaction patterns) would likely improve modeling of mental states that manifest across different timescales (Author, year).…”

“…Gobert, Baker, and Wixon (2015) developed an AAA-based measure for Inq-ITS, a computer-based learning environment to help students develop scientific inquiry skills. In Inq-ITS, students generate hypotheses of scientific phenomena, collect data via simulated experiments embedded in micro-worlds, and evaluate their hypotheses in light of collected data.…”

High school students’ evaluations, plausibility (re) appraisals, and knowledge about topics in Earth science

“…We received assent to conduct our research, as well as parental consent, from 339 participants. However, in a careful inspection of participant data, 40 gave patterned responses (e.g., answering questions in pattern that indicates participants did not read the questions, such as selecting straight 3s or selecting responses that result in a design such as a Christmas tree) that reflected disengagement (Gobert, Baker, & Wixon, ), and as such, were indicators of poor data quality. We eliminated these participants from the study sample, with this disengagement distribution skewed slightly toward SW1 and SW2 (about 44% taught by SW1, n = 18; 33% taught by SW2, n = 13; 8% taught by MA1, n = 3; and 14% taught by MA2, n = 6).…”