Annotating Disengagement for Spoken Dialogue Computer Tutoring

When Does Disengagement Correlate with Learning in Spoken Dialog Computer Tutoring?

“…Prior work suggests that gaming behaviors associated with poorer learning often occur when students lack the knowledge to answer the question [3,2]. 7 Similarly, we hypothesize that students often exhibited linguistic (NLP) gaming in our corpus because the system's limited natural language processing abilities prevented them from eliciting information they needed to answer the question. Together, the results for the NLP-Gaming and Hard Types suggest that if not remediated, disengagement can negatively impact learning when it is caused by questions presupposing knowledge the student doesn't have.…”

“…Our inter-annotator reliability evaluation on a corpus subset showed that our overall disengagement label (0.55 Kappa) and disengagement type labels (0.43 Kappa) can be annotated with moderate reliability on par with prior emotion annotation work [7]. For the current analysis, all student turns in the corpus were manually annotated as summarized below.…”

“…For the current analysis, all student turns in the corpus were manually annotated as summarized below. See [7] for full details of the annotation scheme:…”

“…Student (dis)engagement behaviors have been of particular interest in this research, including displays of gaming, boredom, indifference, (lack of) interest, (low) motivation, curiosity, and flow (e.g. [7,3,9,11,12]). Correlational analyses of student (dis)engagement behaviors in tutoring system corpora have indicated that these behaviors are predictive of learning.…”

Using the Short Form of the MSBS to Assess State Boredom Among Adolescents: Psychometric Evidence by Applying Item Response Theory

“…State boredom is particularly common among adolescents, who often report high levels of boredom (Caldwell, Darling, Payne, & Dowdy, 1999; Schulenberg, Martz, Maslowsky, Patrick, & Staff, 2012; Sharp et al, 2011; Shaw, Caldwell, & Kleiber, 1996; Vodanovich & Kass, 1990). Moreover, some studies have shown that state boredom in adolescence is positively associated with at-risk behaviors (e.g., Biolcati, Mancini, & Trombini, 2018; Hunter & Csikszentmihalyi, 2003; Lin, Lin, & Wu, 2009; Wegner & Flisher, 2009) and negative academic performance (e.g., Craig, Graesser, Sullins, & Gholson, 2004; Forbes-Riley, Litman, & Friedberg, 2011). Thus, it is important to pay attention to how state boredom among youth is measured.…”