The development and analysis of tutorial dialogues in AutoTutor Lite

Abstract

The goal of Intelligent Tutoring Systems (ITS) that interact in natural language is to emulate the benefits a well-trained human tutor provides to students, by interpreting student answers and appropriately responding to encourage elaboration. BRCA Gist is an ITS developed using AutoTutor Lite, a web-based version of AutoTutor. Fuzzy-Trace Theory theoretically motivated the development of BRCA Gist, which engages people in tutorial dialogues to teach them about genetic breast cancer risk. We describe an empirical method to create tutorial dialogues and fine-tune the calibration of BRCA Gist’s semantic processing engine without a team of computer scientists. We created five interactive dialogues centered on pedagogic questions, such as “What should someone do if she receives a positive result for genetic risk of breast cancer?” This method involved an iterative refinement process of repeated testing with different texts, and successively making adjustments to the tutor’s expectations and settings to improve performance. The goal of this method was to enable BRCA Gist to interpret and respond to answers in a manner that best facilitates learning. We developed a method to analyze the efficacy of the tutor’s dialogues. We found that BRCA Gist’s assessment of participants’ answers was highly correlated with the quality of answers found by trained human judges using a reliable rubric. Dialogue quality between users and BRCA Gist, predicted performance on a breast cancer risk knowledge test completed after the tutor. The appropriateness of BRCA Gist feedback also predicted the quality of answers and breast cancer risk knowledge test scores.

Understanding genetic breast cancer risk: Processing loci of the BRCA Gist Intelligent Tutoring System

“…This represents a substantial improvement over the 85% of responses judged to be appropriate in the first iteration of BRCA Gist reported by Wolfe, Widmer, and colleagues (2013) who also found that the percentage of appropriate responses correlated with learning outcomes. Perhaps because, in the current study, BRCA Gist performance approaches ceiling, the percentage of appropriate responses did not predict performance on the assessment instruments, Fs<1.…”

“…Thus, if the tutor is appropriately interpreting verbal inputs from the users, an answer given a high CO score as a measure of semantic similarity should contain more relevant content, as measured by the researcher applying a rubric, than in an answer given a lower CO score. Applying a conditional reliability procedure (Wolfe, Widmer et al, 2013) the two judges achieved .89 agreement with a range of .84 with responses to the question “what should someone do if she receives a negative result for genetic risk of breast cancer,” to .94 for responses to the question "how does breast cancer grow and spread." To assess the effect of the dialogues on learning, we correlated coverage scores with scores on the 52-item multiple-choice test assessing declarative knowledge of genetic risk of breast cancer, gist comprehension scores, and risk assessment percent correct.…”

Efficacy of a Web-Based Intelligent Tutoring System for Communicating Genetic Risk of Breast Cancer

“…An ITS is a computer-based system using artificial intelligence techniques to mimic one-on-one human “tutoring.” 19 Individual human tutoring is arguably the “gold standard” for facilitating deep conceptual understanding, 20,21 and there is evidence that the best ITS's produce gains comparable to human tutors. 22 An ITS is not the same as decision aids such as Gail, Claus, or BRCAPRO.…”

“…ITSs in medicine are rare, 28 and although computer-tailored interventions have demonstrable benefits in areas such as mammography screening, 29 BRCA Gist appears to be the first ITS applied to lay people's medical decision making. 19 …”

AutoTutor and Family: A Review of 17 Years of Natural Language Tutoring

“…The content for BRCA-Gist was authored by researchers at Miami University, making it one of the first AutoTutor descendants authored outside of Memphis. A recent study using BCRA-Gist reported that the AutoTutor Lite's evaluations of student contributions predicted student outcomes on a posttest (Wolfe et al 2013). A second study indicated that AutoTutor Lite outperformed a static web-based tutorial for reducing internal inconsistency (e.g., negating prior assumptions) but did not improve quantitative estimates of risk probabilities (Wolfe et al 2012).…”