Two-way adaptation for robust input interpretation in practical multimodal conversation systems

Pan, Shimei; Shen, Siwei; Zhou, Michelle X.; Houck, Keith

doi:10.1145/1040830.1040849

Cited by 6 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first type of systems handles only independent natural language queries (e.g., [12,22]). The second type of systems processes a user's natural language queries in context (e.g., [1,11]). CENTAUR is an extension of the second type of systems.…”

Section: Related Workmentioning

confidence: 99%

“…Currently, we extend our own natural language query interpretation engine [11], which is capable of handling noisy and ambiguous natural language input, to note interpretation. Our extensions are made in three areas to accommodate the unique characteristics of user note input.…”

Section: Deriving Information Requests In Contextmentioning

confidence: 99%

“…For each identified segment, CENTAUR uses our previous natural language query interpretation engine to derive a set of semantic elements, including data concepts (e.g., Hotel), data attributes, and data constraints [11]. It also infers the semantic relationships between the elements, e.g., the relationship located-in between data concepts Hotel and City.…”

Section: Deriving Multiple Information Requestsmentioning

confidence: 99%

See 2 more Smart Citations

An interactive, smart notepad for context-sensitive information seeking

Zhou

2009

Proceedings of the 14th International Conference on Intelligent User Interfaces

Self Cite

View full text Add to dashboard Cite

We are building an interactive, smart notepad system where users enter brief notes to drive a dynamic information-seeking process. In this paper, we focus on describing our work from two aspects: 1) dynamic interpretation of user notes in context to infer a user's information needs, and 2) automatic generation of data queries to satisfy the inferred user needs. Compared to existing information systems, our work offers three unique contributions. First, our system allows users to focus on what to retrieve instead of how, since users can use brief notes to express their information needs without worrying about specific retrieval details. Second, users can use notes to efficiently request multiple pieces of information at once instead of issuing one query at a time. Third, users can easily update any part of their notes to obtain new or updated information. Whenever a user's notes are modified, our system automatically detects and evaluates all affected note sections to retrieve new or updated information. Our preliminary evaluation shows the promise of this work.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Deriving Information Requests In Contextmentioning

confidence: 99%

Section: Deriving Multiple Information Requestsmentioning

confidence: 99%

See 1 more Smart Citation

An interactive, smart notepad for context-sensitive information seeking

Zhou

2009

Proceedings of the 14th International Conference on Intelligent User Interfaces

Self Cite

View full text Add to dashboard Cite

show abstract

“…While we focus on improving the accuracy of low-level appearance, motion, and acoustic models, we believe our appoach will also be useful in adapting timing and fusion parameters. A different approach to multimodal adaptation is to design a system where the user adapts to the system's recognition capabilities while the system attempts to simultaneously adapt to the user [11]. In the context of audio-visual HMMs, maximum likelihood linear regression (MLLR) has been recently used for speaker adaptation [7].…”

Section: Related Workmentioning

confidence: 99%

Co-Adaptation of audio-visual speech and gesture classifiers

Christoudias

Saenko

Morency

et al. 2006

Proceedings of the 8th International Conference on Multimodal Interfaces

View full text Add to dashboard Cite

The construction of robust multimodal interfaces often requires large amounts of labeled training data to account for cross-user differences and variation in the environment. In this work, we investigate whether unlabeled training data can be leveraged to build more reliable audio-visual classifiers through co-training, a multi-view learning algorithm. Multimodal tasks are good candidates for multi-view learning, since each modality provides a potentially redundant view to the learning algorithm. We apply co-training to two problems: audio-visual speech unit classification, and user agreement recognition using spoken utterances and head gestures. We demonstrate that multimodal co-training can be used to learn from only a few labeled examples in one or both of the audio-visual modalities. We also propose a co-adaptation algorithm, which adapts existing audio-visual classifiers to a particular user or noise condition by leveraging the redundancy in the unlabeled data.

show abstract

“…Recently, research has started on two-way adaptation frameworks whereby both users and systems dynamically adapt to each other's capability and needs during the course of interaction. In addition to helping users to dynamically learn the system's capabilities in context, the approach aims to enhance the overall interpretation capability of a system by learning new user expressions on the fly [33].…”

Section: Error Reduction By Designmentioning

confidence: 99%

Towards a taxonomy of error-handling strategies in recognition-based multi-modal human–computer interfaces

Bourguet

2006

Signal Processing

View full text Add to dashboard Cite

In this paper, we survey the different types of error-handling strategies that have been described in the literature on recognition-based human-computer interfaces. A wide range of strategies can be found in spoken human-machine dialogues, handwriting systems, and multimodal natural interfaces.We then propose a taxonomy for classifying error-handling strategies that has the following three dimensions: the main actor in the error-handling process (machine versus user), the purpose of the strategy (error prevention, discovery, or correction), and the use of different modalities of interaction. The requirements that different error-handling strategies have on different sets of interaction modalities are also discussed. The main aim of this work is to establish a classification that can serve as a tool for understanding how to develop more efficient and more robust multimodal human-machine interfaces.Keywords: recognition-based technology, multimodal interfaces, error-handling, taxonomy, interaction design, interaction robustness Recognition-based technologyMultimodal interaction refers to interaction with the virtual and physical environment through natural modes of communication such as speech, body gestures, handwriting, graphics, or gaze.Unlike keyboards and mice inputs, natural modes of communication usually are non-deterministic, and have to be "recognised" by a recognition system, before they can be passed on to an application.Recent developments in recognition-based technology (e.g. speech and gesture recognition) have opened a myriad of new possibilities for the design and implementation of multimodal applications.Handwriting recognisers, for example, are being used in personal digital assistants (e.g. Paragon's multilingual PenReader software for Pocket PC devices), and speech recognition has made its way into desktop machines (e.g. IBM's ViaVoice TM speech recognition engines). However, designing and

show abstract

Two-way adaptation for robust input interpretation in practical multimodal conversation systems

Cited by 6 publications

References 19 publications

An interactive, smart notepad for context-sensitive information seeking

An interactive, smart notepad for context-sensitive information seeking

Co-Adaptation of audio-visual speech and gesture classifiers

Towards a taxonomy of error-handling strategies in recognition-based multi-modal human–computer interfaces

Contact Info

Product

Resources

About