Visual perception, language and gesture: A model for their understanding in multimodal dialogue systems

Abstract

To cite this version:Frédéric Landragin. Visual perception, language and gesture: A model for their understanding in multimodal dialogue systems. Signal Processing, Elsevier, 2006, 86 (12) AbstractThe way we see the objects around us determines speech and gestures we use to refer to them. The gestures we produce structure our visual perception. The words we use have an influence on the way we see. In this manner, visual perception, language and gesture present multiple interactions between each other. The problem is global and has to be tackled as a whole in order to understand the complexity of reference phenomena and to deduce a formal model. This model may be useful for any kind of human-machine dialogue system that focuses on deep comprehension. We show how a referring act takes place into a contextual subset of objects. This subset is called 'reference domain' and is implicit. It can be deduced from a lot of clues. Among these clues are those which come from the visual context and those which come from the multimodal utterance. We present the 'multimodal reference domain' model that takes these clues into account and that can be exploited in a multimodal dialogue system when interpreting.

Visual environment and interlocutors in situated dialogue

“…While up to 57 potential referents were on the board at any given time, speakers and addressees only considered those that had been mentioned recently, that were relevant to the task, and were in close physical proximity to the last mentioned object. Similar task-based constraints have been found to constrain referring in other task--related conversations (Beun & Cremers, 1998; also see Landragin, 2006), suggesting these effects are not limited to the particular task used in this study. Lexical competition during spoken word recognition can be attenuated by other constraints as well, including semantic information (Barr, 2008), talker preferences (e.g., if one talker always says candy, and a different talker always says candle, Creel, et al, 2008), and structural priming of verbs (Thothathiri & Snedeker, 2008).…”

Experimental Approaches to Referential Domains and the On-Line Processing of Referring Expressions in Unscripted Conversation

“…In such settings, the referential domain, or the domain of interpretation for the referring expression, can be easily identified as the set of objects shown in the display. Identification of the referential domain in unscripted conversation is likely to be significantly more complex [41], particularly when the set of potential referents is large, when the interlocutors have different perspectives on the referential domain, and when the potential discourse referents have different affordances. For example, Chambers and colleagues [42] tested whether the affordances of a potential referent and their consistency with a spatial preposition guided listeners' interpretation of instructions.…”

Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

“…There are several areas of research that are relevant to our work, the first one being the vast literature on multimodality -we will just focus on multimodal referring expressions in this paper. As is well known (Sinclair, 1992;Kehler, 2000;Goldin-Meadow, 2005;Landragin, 2006;Navarretta, 2011), in natural dialogue, the antecedents of linguistic referring expressions are often introduced via gestures; for example in our environment, the user can point to a street intersection on a map yet never have mentioned it earlier. Crucially from a computational point of view, including hand gestures information improves the performance of the reference resolution module (Eisenstein and Davis, 2006;Baldwin et al, 2009).…”