Miscommunication Detection and Recovery in Situated Human–Robot Dialogue

Abstract

Even without speech recognition errors, robots may face difficulties interpreting natural-language instructions. We present a method for robustly handling miscommunication between people and robots in task-oriented spoken dialogue. This capability is implemented in TeamTalk, a conversational interface to robots that supports detection and recovery from the situated grounding problems of referential ambiguity and impossible actions. We introduce a representation that detects these problems and a nearest-neighbor learning algorithm that selects recovery strategies for a virtual robot. When the robot encounters a grounding problem, it looks back on its interaction history to consider how it resolved similar situations. The learning method is trained initially on crowdsourced data but is then supplemented by interactions from a longitudinal user study in which six participants performed navigation tasks with the robot. We compare results collected using a general model to user-specific models and find that user-specific models perform best on measures of dialogue efficiency, while the general model yields the highest agreement with human judges. Our overall contribution is a novel approach to detecting and recovering from miscommunication in dialogue by including situated context, namely, information from a robot's path planner and surroundings. CCS Concepts: • Computing methodologies → Discourse, dialogue and pragmatics; • Humancentered computing → Natural language interfaces;

Grounding behaviours with conversational interfaces: effects of embodiment and failures

“…As interactions with conversational agents are becoming increasingly common, it is more likely that people will encounter failures with these systems. It is therefore important to investigate how people's behaviours are affected when system failures cause misunderstandings [10,57]. Researchers have however reported mixed results in the effects of robot failure on people's behaviour and perception of the robot.…”

A scoping review on the use, processing and fusion of geographic data in virtual assistants

“…Finally, improving the conversation and understanding the subtle emotions in the communication between humans and assistants (Portela & Granell‐Canut, 2017) is a primary research line regardless of the target application domain. In this context, Hamzei et al (2020) explore structural patterns of questions related to places and their human‐generated responses, and Marge and Rudnicky (2019) investigate conversational interfaces between a human operator giving navigational instructions to a team of robots.…”

Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

“…For instance, users appear to pay less attention when selecting example objects, clicking Shake Head, implicitly labeling all shown objects as negative examples, even when positive ones exist in the set. It may help to infer user truthfulness during dialog (Vinanzi, Patacchiola, Chella, & Cangelosi, 2019) as an auxiliary goal, or to use user-specific dialog policies (Doering et al, 2019) to develop rapport (Marge & Rudnicky, 2019).…”