The roles of haptic-ostensive referring expressions in cooperative, task-based human-robot dialogue

Abstract

Generating referring expressions is a task that has received a great deal of attention in the natural-language generation community, with an increasing amount of recent effort targeted at the generation of multimodal referring expressions. However, most implemented systems tend to assume very little shared knowledge between the speaker and the hearer, and therefore must generate fully-elaborated linguistic references. Some systems do include a representation of the physical context or the dialogue context; however, other sources of contextual information are not normally used. Also, the generated references normally consist only of language and, possibly, deictic pointing gestures.When referring to objects in the context of a task-based interaction involving jointly manipulating objects, a much richer notion of context is available, which permits a wider range of referring options. In particular, when conversational partners cooperate on a mutual task in a shared environment, objects can be made accessible simply by manipulating them as part of the task. We demonstrate that such expressions are common in a corpus of human-human dialogues based on constructing virtual objects, and then describe how this type of reference can be incorporated into the output of a humanoid robot that engages in similar joint construction dialogues with a human partner.

Design Principles for Safety in Human-Robot Interaction

“…Addition-HUMAN 1: And I'll get this HUMAN 1: And then the red one HUMAN 2: 'Kay I've got the yellow HUMAN 1: Cool Fig. 2 Example of a real dialogue between two humans during a joint construction task ally, as Foster et al showed in [12], many spoken expressions that refer to objects in the world can only be understood together with the gestures that accompany the spoken part of the message. For example, Foster et al present a real dialogue from a joint construction task, in which two humans assemble tangram models together.…”

How Quickly Should a Communication Robot Respond? Delaying Strategies and Habituation Effects

“…Note that this slowness is not unique to our robot. For instance, Foster et al report that it takes a few seconds or more for their robot to react autonomously to users at the moment [10]. Thus, finding the upper boundary for which users will wait is important.…”

Generating a Novel Dataset of Multimodal Referring Expressions

“…Studies in this area have a long history in computational linguistics/semantics (e.g., Claassen (1992); Krahmer and van der Sluis (2003)), human-robot interaction (e.g., Kelleher and Kruijff (2006);Foster et al (2008)), and computational and human discourse studies (e.g., Bortfeld and Brennan (1997); Funakoshi et al (2004); Viethen and Dale (2008)). Following these, we seek to build models for generating, recognizing, and classifying referring expressions that are both natural and useful to the human interlocutors of computational dialogue systems.…”