Goal-oriented multimedia dialogue with variable initiative

Reinforcement Learning with Immediate Rewards and Linear Hypotheses

“…One module in a typical architecture for such a system takes as input some abstract representation of an idea to convey to the user and must output a way to express that idea. The output module in the Pascal tutor developed by members of the Duke Voice and Natural Language Lab (see [13]) takes as input a prolog statement, and must output a message that is a mix of speech, text, and graphics. The most interesting subproblem faced by this module was identifying part of a program (for example, choosing between saying "there is something wrong with the fourth word on the third line", saying "there is something wrong with the highlighted region" while highlighting, and so on), so we focus on this subproblem for the remainder of this section.…”

“…Comments. We have implemented this system, using a variant of CAU, and integrated it into an existing Pascal tutor [13]. The tutor as a whole works in real time, interacting with the user using speech recognition and generated speech, graphics, and text as described in this paper, as well as by observing the user's current program.…”

“…A reasonable goal is to construct messages that are easy for the user to understand. In Appendix A we describe a grammar-based method for generating multimodal messages that we have implemented and integrated into an existing tutoring system [13]. The system adapts to the preferences of the user using reinforcement learning from immediate rewards and a linear model that is motivated by a coarse user model.…”

PartsID

“…In addition to the works mentioned above, there have been several classic projects in the area of natural language dialogue like TRAINS/TRIPS project at Rochester (Allen et al, 1989(Allen et al, , 1995(Allen et al, , 1996, Duke's Circuit-Fixit-Shoppe and Pascal Tutoring System (Biermann et al, 1997;1995), etc. While the Circuit-Fixit-Shoppe system helps users fix a circuit through a dialogue with the system, the TRIPS and the TRAINS projects allow users to plan their itineraries through dialogue.…”

Mechanisms for mixed-initiative human-computer collaborative discourse

“…Currently, two spoken-dialogue human-computer systems are being developed using the underlying algorithms described in this paper. The Duke Programming Tutor instructs introductory computer science students how to write simple Pascal programs by providing multiple modes of input and output (voice/text/graphics) (Bierman et al, 1996). The Advanced Maintenance Assistant and Trainer (AMAT) currently being developed by Research Triangle Institute for the U.S. Army allows a maintenance trainee to converse with a computer assistant in the diagnosis and repair of a virtual MIA1 tank.…”