Referring to objects with spoken and haptic modalities

Landragin, Frédéric; Bellalem, Nadia; Romary, Laurent

doi:10.1109/icmi.2002.1166976

Cited by 11 publications

(7 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It was an opportunity for us to test the relevance of reference domains for haptic humanmachine interaction. As it is described in [20], we imagined tactile reference domains. That was a means for proving the interest and the flexibility of our multimodal reference domain model for various interaction paradigms.…”

Section: Designing Multimodal Systemsmentioning

confidence: 99%

Visual perception, language and gesture: A model for their understanding in multimodal dialogue systems

Landragin

2006

Signal Processing

View full text Add to dashboard Cite

To cite this version:Frédéric Landragin. Visual perception, language and gesture: A model for their understanding in multimodal dialogue systems. Signal Processing, Elsevier, 2006, 86 (12) AbstractThe way we see the objects around us determines speech and gestures we use to refer to them. The gestures we produce structure our visual perception. The words we use have an influence on the way we see. In this manner, visual perception, language and gesture present multiple interactions between each other. The problem is global and has to be tackled as a whole in order to understand the complexity of reference phenomena and to deduce a formal model. This model may be useful for any kind of human-machine dialogue system that focuses on deep comprehension. We show how a referring act takes place into a contextual subset of objects. This subset is called 'reference domain' and is implicit. It can be deduced from a lot of clues. Among these clues are those which come from the visual context and those which come from the multimodal utterance. We present the 'multimodal reference domain' model that takes these clues into account and that can be exploited in a multimodal dialogue system when interpreting.

show abstract

Section: Designing Multimodal Systemsmentioning

confidence: 99%

Visual perception, language and gesture: A model for their understanding in multimodal dialogue systems

Landragin

2006

Signal Processing

View full text Add to dashboard Cite

show abstract

“…The referring expressions can make use of the task context and the state of the workspace in addition to the history of the discourse and the current visual state. As well, since both partners are able to-and, indeed, must-manipulate objects in the world as part of the task, another possible type of reference becomes possible: haptic-ostensive reference [14], that is, referring to an object by manipulating it in the world.…”

Section: Haptic-ostensive Reference In a Shared Workpacementioning

confidence: 99%

The roles of haptic-ostensive referring expressions in cooperative, task-based human-robot dialogue

Foster

Bard

Guhe

et al. 2008

Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction

View full text Add to dashboard Cite

Generating referring expressions is a task that has received a great deal of attention in the natural-language generation community, with an increasing amount of recent effort targeted at the generation of multimodal referring expressions. However, most implemented systems tend to assume very little shared knowledge between the speaker and the hearer, and therefore must generate fully-elaborated linguistic references. Some systems do include a representation of the physical context or the dialogue context; however, other sources of contextual information are not normally used. Also, the generated references normally consist only of language and, possibly, deictic pointing gestures.When referring to objects in the context of a task-based interaction involving jointly manipulating objects, a much richer notion of context is available, which permits a wider range of referring options. In particular, when conversational partners cooperate on a mutual task in a shared environment, objects can be made accessible simply by manipulating them as part of the task. We demonstrate that such expressions are common in a corpus of human-human dialogues based on constructing virtual objects, and then describe how this type of reference can be incorporated into the output of a humanoid robot that engages in similar joint construction dialogues with a human partner.

show abstract

“…For an application scenario within this project, music selection and exploration on a portable device was selected, with multimodal interaction capabilities [4,5]. Songs can be selected from a multidimensional database in which they are labeled according to different dimensions, such as genre (for example jazz), time-period (seventies), artist (Madonna) and mood Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.…”

Section: Introductionmentioning

confidence: 99%

User walkthrough of multimodal access to multidimensional databases

Esch-Bussemakers¹,

Cremers²

2004

Proceedings of the 6th International Conference on Multimodal Interfaces

View full text Add to dashboard Cite

This paper describes a user walkthrough that was conducted with an experimental multimodal dialogue system to access a multidimensional music database using a simulated mobile device (including a technically challenging four-PHANToM-setup). The main objectives of the user walkthrough were to assess user preferences for certain modalities (speech, graphical and haptictactile) to access and present certain types of information, and for certain search strategies when searching and browsing a multidimensional database. In addition, the project aimed at providing concrete recommendations for the experimental setup, multimodal user interface design and evaluation. The results show that recommendations can be formulated both on the use of modalities and search strategies, and on the experimental setup as a whole, including the user interface. In short, it is found that haptically enhanced buttons are preferred for navigating or selecting and speech is preferred for searching the database for an album or artist. A 'direct' search strategy indicating an album, artist or genre is favorable. It can be concluded that participants were able to look beyond the experimental setup and see the potential of the envisioned mobile device and its modalities. Therefore it was possible to formulate recommendations for future multimodal dialogue systems for multidimensional database access.

show abstract

Referring to objects with spoken and haptic modalities

Cited by 11 publications

References 6 publications

Visual perception, language and gesture: A model for their understanding in multimodal dialogue systems

Visual perception, language and gesture: A model for their understanding in multimodal dialogue systems

The roles of haptic-ostensive referring expressions in cooperative, task-based human-robot dialogue

User walkthrough of multimodal access to multidimensional databases

Contact Info

Product

Resources

About