A practical methodology for the evaluation of spoken language systems

Machine Conversations

“…They will be manageable if interruption points can be discerned in the real-time using acoustic information (Nakatani and Hirschberg, 1994 Parsing Utterances Including Self-Repairs It is relatively easy to evaluate technologies such as morphological analysis and information retrieval in objecive and empirical terms, because unique solutions can be defined for such tasks. Such an evaluation will be almost necessarily a blackbox evaluation, such as in ATIS (Boisen and Bates, 1992), TREC (Harman, 1995), MUC (MUC, 1991), and so forth. In order to advance researches on dialogue systems, there should be some empirical method for evaluating them.…”

Improving the efficacy of automated sign language practice tools

“…Boisen and Bates developed a methodology based on the collective experiences of BBN's participation in the DARPA projects [15]. Their methodology analyzed many domain specific evaluation methods to create a general framework to characterize the evaluation of dialogue systems.…”

“…The recognition work, combined with the work in this chapter, are ultimately designed to create a complete running CopyCat system for deployment and testing with live children. There has been a great amount of research on evaluating live system testing for dialogue systems that have been designed using Wizard of Oz systems [15,66,72,114,148]. There is a tension between domain specific criteria, intermediary evaluation and metrics, human judgement, and the input / output mapping of the final system.…”

“…Boisen and Bates's evaluation architecture for dialogue systems (Image used from Boisen and Bates "A practical methodology for the evaluation of spoken language systems" with permission [15]) . .…”

Evaluating natural language processing systems

“…Other evaluations in the same tradition include the ATIS [2] and TREC [10] evaluations, the first in the domain of database query, emphasizing a spoken language component, the second in the domain of text retrieval. A third set of evaluations-the MUC evaluations of fact extraction systems-is reported by Cowie and Lehnert in this issue and is reported in detail in [18].The test material for these other evaluations in the ARPA tradition is, however, critically different.…”