Multi-Modal Definite Clause Grammar

Shimazu, Hideo; Arita, Seiko; Takashima, Yuki

doi:10.3115/991250.991281

Cited by 8 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous research on a multimodal dialogue system focused on finding the relationship between a pointing gesture and a deictic expression [Bolt 1980;Neal et al 1988;Salisbury et al 1990;Shimazu et al 1994;Shimazu and Takashima 1996] and on mapping a predefined symbol to a simple command [Johnston et al 1997]. None of them, however, suggested methods for resolving deictic expressions where pointing gestures are omitted; e.g., 빨간 거 (the red item) in Utterance (8).…”

Section: 빨간 거 얼마죠?mentioning

confidence: 97%

Resolution of referring expressions in a Korean multimodal dialogue system

Kim

Seo

2003

ACM Transactions on Asian Language Information Processing

View full text Add to dashboard Cite

__________________________________________________________________________________________Referring expressions in multimodal dialogues have different aspects compared to those in language-only dialogues. They often refer to the items signified by either a gesture or visual means. In this article we classify referring expressions into two types (i.e., a deictic reference and an anaphoric reference), and propose two general methods to resolve these referring expressions. One method is a simple mapping algorithm that can find items referred with/without pointing gestures on a screen. The other is the centering algorithm with a dual cache model, to which Walker's centering algorithm is extended for a multimodal dialogue system. The extended algorithm is appropriate for resolving various anaphoric references in a multimodal dialogue. In the experiments, the proposed system correctly resolved 376 out of 405 referring expressions in 40 dialogues (0.54 referring expressions per utterance) showing 92.84 percent correctness.

show abstract

Section: 빨간 거 얼마죠?mentioning

confidence: 97%

Resolution of referring expressions in a Korean multimodal dialogue system

Kim

Seo

2003

ACM Transactions on Asian Language Information Processing

View full text Add to dashboard Cite

show abstract

“…To date, a number of academic approaches have proposed reliable methodologies to design multimodal interaction, e.g. architectural approach [5], frame-based approach [8], statistical approach [11], and grammatical approach [13]. The common limitation of these approaches, however, is that they have addressed only specific cases of multimodality and failed to generalise their methodologies to a range of applications.…”

Section: Fsm Modellingmentioning

confidence: 99%

Usability Framework for the Design and Evaluation of Multimodal Interaction

Chang

Bourguet

2008

Electronic Workshops in Computing

View full text Add to dashboard Cite

The design and evaluation of multimodal interaction is difficult. For designers in industry, developing multimodal interaction systems is a big challenge. Although past researches have presented various methodologies, they have addressed only specific cases of multimodality and failed to generalise their methodologies to a range of applications. In this paper, we present a usability framework for the design and evaluation of multimodal interaction. First, in the early phase of multimodality design, elementary multimodal commands are elicited using traditional usability techniques. Second, based on the CARE (Complementarity, Assignment, Redundancy, and Equivalence) properties and the FSM (Finite State Machine) formalism, the original set of elementary commands is automatically expanded to form a more comprehensive set of multimodal commands. Third, this new set of multimodal commands is evaluated in two ways: user-testing and errorrobustness evaluation. This framework acts as a structured and general methodology both for designing and evaluating multimodal interaction. We expect that it will help designers to produce more usable multimodal systems.

show abstract

“…Using grammars for the processing of multimodal cues has been proposed in different variations. The first who reported a grammatical framework for a multimodal interface were Shimazu, Arita, and Takashima [1]. They developed a Multimodal Definite Clause Grammar (MM-DCG) that was able to handle an arbitrary number of modes and stored temporal information inside the grammar.…”

Section: Related Workmentioning

confidence: 99%

Integrating Multimodal Cues Using Grammar Based Models

Giuliani

Knoll

2007

Universal Access in Human-Computer Interaction. Ambient Interaction

View full text Add to dashboard Cite

Multimodal systems must process several input streams efficiently and represent the input in a way that allows the establishment of connections between modalities. This paper describes a multimodal system that uses Combinatory Categorial Grammars to parse several input streams and translate them into logical formulas. These logical formulas are expressed in Hybrid Logic, which is very suitable for multimodal integration because it can represent temporal relationships between modes in an abstract way. This level of abstraction makes it possible to define rules for multimodal processing in a straightforward way.

show abstract

Multi-Modal Definite Clause Grammar

Cited by 8 publications

References 8 publications

Resolution of referring expressions in a Korean multimodal dialogue system

Resolution of referring expressions in a Korean multimodal dialogue system

Usability Framework for the Design and Evaluation of Multimodal Interaction

Integrating Multimodal Cues Using Grammar Based Models

Contact Info

Product

Resources

About