Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system

Abstract

We describe the design and evaluation of two different dynamic student uncertainty adaptations in wizarded versions of a spoken dialogue tutoring system. The two adaptive systems adapt to each student turn based on its uncertainty, after an unseen human "wizard" performs speech recognition and natural language understanding and annotates the turn for uncertainty. The design of our two uncertainty adaptations is based on a hypothesis in the literature that uncertainty is an "opportunity to learn"; both adaptations use additional substantive content to respond to uncertain turns, but the two adaptations vary in the complexity of these responses. The evaluation of our two uncertainty adaptations represents one of the first controlled experiments to investigate whether substantive dynamic responses to student affect can significantly improve performance in computer tutors. To our knowledge we are the first study to show that dynamically responding to uncertainty can significantly improve learning during computer tutoring. We also highlight our ongoing evaluation of our uncertainty-adaptive systems with respect to other important performance metrics, and we discuss how our corpus can be used by the wider computer speech and language community as a linguistic resource supporting further research on effective affect-adaptive spoken dialogue systems in general.

Towards Improving (Meta)cognition by Adapting to Student Uncertainty in Tutorial Dialogue

“…Second, Random was changed so that ITSPOKE-AUTO randomly remediated after only CnonU answers (non-impasse states). We changed this condition because in ITSPOKE-WOZ neither wizarded experimental condition outperformed Random , 2011b ; we hypothesized this was because CU impasses were sometimes adapted to in Random . Full details of the ITSPOKE-AUTO system, including a performance analysis of the speech and language processing components and their impact on the learning results, are presented elsewhere (Forbes-Riley & Litman, 2010, 2011a .…”

“…As shown, in each case, ITSPOKE2 provides feedback responding to the speci fi c impasse type of the answer and then provides the same additional content responding to the impasse, but the dialogue act used to present Finally, the Random control condition treated a percentage of random correct answers as incorrect, to control for the additional content in the experimental conditions. The motivation for and further details of each experimental condition are discussed in detail elsewhere , 2011b .…”

“…First, the Complex experimental condition was removed. We removed this condition as only Simple yielded learning improvements for ITSPOKE-WOZ , 2011b . Second, Random was changed so that ITSPOKE-AUTO randomly remediated after only CnonU answers (non-impasse states).…”

“…1 The corpora were collected in our prior experiments evaluating the utility of enhancing ITSPOKE to respond to impasses involving student uncertainty over and above correctness, in wizarded , 2011b and fully automated (Forbes-Riley & Litman, 2011a ) conditions. The target audience for ITSPOKE are novices, i.e., college students who have never taken college-level physics.…”

“…In our experimental conditions, the system provided additional knowledge at places of uncertainty; in the control conditions, the system either did not provide this knowledge, or provided such knowledge randomly. In a fi rst experiment we used a wizarded form of our system, where uncertainty and correctness were manually annotated in real time by a human "wizard" , 2011b . Our results demonstrated that responding to student uncertainty, over and above correctness, did indeed lead to performance improvements along cognitive dimensions.…”