Field trial evaluations of two different information inquiry systems

Billi, R.; Castagneri, Giuseppe; Danieli, Morena

doi:10.1016/s0167-6393(97)00041-1

Cited by 18 publications

(10 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, telephone interactions provide a very challenging environment for speech recognition equipment. For the Dialogos system which answers inquiries about Italian Railway train schedules, (Billi, Castagneri, and Danieli, 1996) report only 68.2% word accuracy for the system in 96 dialogs. In spite of this Dialogos still understood 81.6% of all sentences, a promising result.…”

Section: Recognition Ratementioning

confidence: 99%

Performance measures for the next generation of spoken natural language dialog systems

Smith

1997

Interactive Spoken Dialog Systems on Bringing Speech and NLP Together in Real Applications - ISDS '97

View full text Add to dashboard Cite

1Improved Performance in Spoken Natural Language Dialog SystemsSince approximately the mid 1980's, technology has been adequate (if not ideal) for researchers to construct spoken natural language dialog systems (SNLDS) in order to test theories of natural language processing and to see what machines were capable of based on current technological limits. Over the course of time, a few systems have been constructed in sufficient detail and robustness to enable some evaluation of the systems. For the most part, these systems were greatly limited by the available speech recognition technology. Continuous speech systems required speaker dependent training and restricted vocabularies, but still had such a large number of misrecognitions that this tended to be the limiting factor in the success of the system. For example, testing in 1991 of the Circuit Fix-It Shop of (Smith, Hipp, and Biermann, 1995) required an experimenter to remain in the room in order to notify the user when misrecognitions occurred. Fortunately, speech recognition capabilities are improving, and systems are being constructed that allow individuals to walk up and use them after a brief orientation. One example is the TRAINS system of (Allen et al., 1995) that was demonstrated at the 1995 ACL conference, where people just sat down and used the system after a brief set of instructions were given to them by the demonstrator. Another example is the current system under development at Duke University that serves as a tutor for liberal arts students learning the basics of Pascal programming. In this system, the machine itself explains how to use it. More thorough and challenging methods of evaluation are now feasible. This paper proposes some measures for evaluation based on a retrospective look at measures used in the past, analyzing their relevance in today's environment.For the future, expect measurements of speech recognition performance and basic utterance understanding to remain important, but there should also be more emphasis on measuring robustness and measuring the utility of domain-independent knowledge about dialog. Furthermore, we should expect realtime response from evaluated systems, a sharp reduction in the amount of specialized training for using systems, and the use of longitudinal studies to see how user behavior evolves. Fundamentals in Evaluation Linguistic CoverageA forward looking view of evaluation is offered by (Whittaker and Stenton, 1989). It is forward looking in the sense that they investigated issues in evaluation independent of building a system. Their perspective was not based on a specific SNLDS, but a general analysis of the issue of evaluation. Their main point was that evaluation needed to be placed within the context of a system's use. Consequently, they used a Wizard of Oz study in an information retrieval environment (e.g., database query) in order to identify the types of natural language inputs a typical user would use in order to gain access to needed information. Their analysis identified the following requireme...

show abstract

Section: Recognition Ratementioning

confidence: 99%

Performance measures for the next generation of spoken natural language dialog systems

Smith

1997

Interactive Spoken Dialog Systems on Bringing Speech and NLP Together in Real Applications - ISDS '97

View full text Add to dashboard Cite

show abstract

“…With the growing availability of various content provided over state-of-the-art digital media is speech recognition becoming one of the main core technologies (Billi et al, 1997;Žgank et al, 2002;Gupta et al, 2000;Sket et al, 2002). Its task is to minimize the needed effort to access the particular part of content.…”

Section: Introductionmentioning

confidence: 99%

Modeling of Filled Pauses and Onomatopoeas for Spontaneous Speech Recognition

Žgank¹,

Maučec²

2010

Advances in Speech Recognition

View full text Add to dashboard Cite

“…These systems are computer programs developed to provide specific services using speech, for example airplane travel information , train travel information (Billi et al 1997), weather forecasts (Zue et al 2000;Nakano et al 2001), fast food ordering (Seto et al 1994;López-Cózar et al 1997), call routing (Lee et al 2000) or directory assistance (Kellner et al 1997).…”

Section: Introductionmentioning

confidence: 99%