Automatic detection of learner’s affect from conversational cues

Abstract

We explored the reliability of detecting a learner's affect from conversational features extracted from interactions with AutoTutor, an intelligent tutoring system (ITS) that helps students learn by holding a conversation in natural language. Training data were collected in a learning session with AutoTutor, after which the affective states of the learner were rated by the learner, a peer, and two trained judges. Inter-rater reliability scores indicated that the classifications of the trained judges were more reliable than the novice judges. Seven data sets that temporally integrated the affective judgments with the dialogue features of each learner were constructed. The first four datasets corresponded to the judgments of the learner, a peer, and two trained judges, while the remaining three data sets combined judgments of two or more raters. Multiple regression analyses confirmed the hypothesis that dialogue features could significantly predict the affective states of boredom, confusion, flow, and frustration. Machine learning experiments indicated that standard classifiers were moderately successful in discriminating the affective states of boredom, confusion, flow, frustration, 123 46 S. K. D'Mello et al. and neutral, yielding a peak accuracy of 42% with neutral (chance = 20%) and 54% without neutral (chance = 25%). Individual detections of boredom, confusion, flow, and frustration, when contrasted with neutral affect, had maximum accuracies of 69, 68, 71, and 78%, respectively (chance = 50%). The classifiers that operated on the emotion judgments of the trained judges and combined models outperformed those based on judgments of the novices (i.e., the self and peer). Follow-up classification analyses that assessed the degree to which machine-generated affect labels correlated with affect judgments provided by humans revealed that human-machine agreement was on par with novice judges (self and peer) but quantitatively lower than trained judges. We discuss the prospects of extending AutoTutor into an affect-sensing ITS.

Exploring Students’ Engagement Patterns in SPOC Forums and their Association with Course Performance

“…In this case, it is difficult for instructors to identify the struggling, confused and even frustrated students. Previous research has indicated that positive emotion helps to promote students' learning interests and engagement levels (Altrabsheh, Cocea, & Fallahkhair, 2015;D'Mello et al, 2008), and students with an upsurge of emotion may have higher motivation to accomplish their learning goals. Ramesh, Goldwasser, Huang, Daumé III, & Getoor (2013) incorporated the positive/negative score of post content into engagement metrics to distinguish between disengaged and engaged learners in MOOC forums.…”

Modeling User Affect from Causes and Effects

“…We try to capture potentially overlapping emotions, adding an additional level of complexity to the modeling task. Second, [12] targets longerterm states that some researchers may classify as moods, i.e., states that are less specific than simple emotions, less likely to be triggered by a particular stimulus, and lasting [10]. We see these longer-term affective states as being complementary to the more instantaneous emotions we focus on, as we discuss in a later section.…”

“…In contrast, most work on affect recognition has focused on detecting one specific emotion (e.g., [4][5][6]), lower-level affective measures of valence and arousal (e.g., [7,9]) or overall emotional predisposition over a complete interaction (e.g., [10,11]). One exception is the work by D'Mello et al, [12], which used dialogue features as predictors of student's boredom, confusion, flow and frustration during interaction with a dialogue-based tutoring system. There are three main differences between this work and ours.…”

“…There are three main differences between this work and ours. First, in [12] the target emotions are treated as mutually exclusive, which they mostly are, with the exception perhaps of confusion and frustration. We try to capture potentially overlapping emotions, adding an additional level of complexity to the modeling task.…”

“…We see these longer-term affective states as being complementary to the more instantaneous emotions we focus on, as we discuss in a later section. Finally, the approach in [12] does not include an explicit representation of causes of affect, thus providing less information than our approach for an agent to decide how to best deal with the student's emotions.…”