A proposal for SLS evaluation

Boisen, Sean; Ramshaw, Lance; Ayuso, Damaris; Bates, Madeleine

doi:10.3115/1075434.1075457

Cited by 12 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It may be conjectured that this is due to the fact that most of the critical key words and phrases are very common in the training data for the task and are therefore well modeled both acoustically and in the statistical language models used by the recognizers. (12)(13)(14)(15). However, the model depends on having a natural language grammar that accurately models the speech being recognized.…”

Section: Robustness To Recognition Errorsmentioning

confidence: 99%

See 1 more Smart Citation

Integration of speech with natural language understanding.

Moore¹

1995

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

The integration of speech recognition with natural language understanding raises issues of how to adapt natural language processing to the characteristics of spoken language; how to cope with errorful recognition output, including the use of natural language information to reduce recognition errors; and how to use information from the speech signal, beyond just the sequence of words, as an aid to understanding. This paper reviews current research addressing these questions in the Spoken Language Program sponsored by the Advanced Research Projects Agency (ARPA). I begin by reviewing some of the ways that spontaneous spoken language differs from standard written language and discuss methods of coping with the difficulties of spontaneous speech. I then look at how systems cope with errors in speech recognition and at attempts to use natural language information to reduce recognition errors. Finally, I discuss how prosodic information in the speech signal might be used to improve understanding.The goal of integrating speech recognition with natural language understanding is to produce spoken-languageunderstanding systems-that is, systems that take spoken language as their input and respond in an appropriate way depending on the meaning of the input. Since speech recognition (1) aims to transform speech into text, and naturallanguage-understanding systems (2) aim to understand text, it might seem that spoken-language-understanding systems could be created by the simple serial connection of a speech recognizer and a natural-language-understanding system. This naive approach is less than ideal for a number of reasons, the most important being the following: * Spontaneous spoken language differs in a number of ways from standard written language, so that even if a speech recognizer were able to deliver a perfect transcription to a natural-language-understanding system, performance would still suffer if the natural language system were not adapted to the characteristics of spoken language.* Current speech recognition systems are far from perfect transcribers of spoken language, which raises questions about how to make natural-language-understanding systems robust to recognition errors and whether higher overall performance can be achieved by a tighter integration of Apeech recognition and natural language understanding.* Spoken language contains information that is not necessarily represented in written language, such as the distinctions between words that are pronounced differently but spelled the same, or syntactic and semantic information that is encoded prosodically in speech. In principle it should be possible to extract this information to solve certain understanding problems more easily using spoken input than using a simple textual transcription of that input.This paper looks at how these issues are being addressed in current research in the ARPA Spoken Language Program. COPING WITH SPONTANEOUS SPOKEN LANGUAGE Language Phenomena in Spontaneous SpeechThe participants in the ARPA Spoken Language Program hav...

show abstract

Section: Robustness To Recognition Errorsmentioning

confidence: 99%

“…Word lattice parsing was explored by BBN (12,13) in the early stages of the current ARPA effort. In this approach the recognizer produces a set of word hypotheses, with an acoustic score for each potential pair of start and end points for each possible word.…”

Section: Word Lattice Parsingmentioning

confidence: 99%

Integration of speech with natural language understanding.

Moore¹

1995

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…For the sake of automatic evaluation, then, a canonical reference answer (the minimum "right answer") is developed for each evaluable query in the training set. The content of this reference answer is determined both by domainindependent linguistic principles (Boisen et al, 1989) and domain-specific stipulation. The language used to express the answers for the ATIS domain is presented in Appendix A.…”

Section: Reference Answersmentioning

confidence: 99%

A practical methodology for the evaluation of spoken language systems

Boisen¹,

Bates²

1992

Proceedings of the Third Conference on Applied Natural Language Processing -

Self Cite

View full text Add to dashboard Cite

“…This is, of course, often not true in spontaneous goal-directed utterances, since one query may create a context for another, and the full context is required to answer (e.g., "Show me the flights ... ", 'Which of THEM ..."). Various means of extending this methodology for evaluating context-dependent queries have been proposed, and some of them have been implemented in the ATIS evaluations (Boisen et al (1989), Hirschman et al (1990, Bates and Ayuso (1991), Pallett (1991)). …”

Section: Discourse Contextmentioning

confidence: 99%

“…The content of this reference answer is determined both by domainindependent linguistic principles (Boisen et al, 1989) and domain-specific stipulation. The language used to express the answers for the ATIS domain is presented in Appendix A.…”

Section: Reference Answersmentioning

confidence: 99%