Robot perception errors and human resolution strategies in situated human–robot dialogue

Understanding and Resolving Failures in Human-Robot Interaction: Literature Review and Model Development

“…The quality of error recovery and communication strategies have been evaluated using various performance metrics, including whether users managed to resolve the problems (Spexard et al, 2008 ), attribution of blame (Kim and Hinds, 2006 ), the frequency of use of recovery feature (Spexard et al, 2008 ), the number of error-free user interactions (Gieselmann and Ostendorf, 2007 ; Knepper et al, 2015 ), time per repair (Rosenthal et al, 2012 ; Knepper et al, 2015 ; van der Woerdt and Haselager, 2017 ), time until task completion (De Visser and Parasuraman, 2011 ; Rosenthal et al, 2012 ; Schütte et al, 2017 ), user comfort (Engelhardt and Hansson, 2017 ), user satisfaction (Gieselmann and Ostendorf, 2007 ; Shiomi et al, 2013 ), task performance and completion (Gieselmann and Ostendorf, 2007 ; De Visser and Parasuraman, 2011 ; Desai et al, 2013 ; Salem et al, 2013 ; Knepper et al, 2015 ; Brooks, 2017 ; Schütte et al, 2017 ), workload (Brooks, 2017 ), confidence (De Visser and Parasuraman, 2011 ; Brooks, 2017 ), comprehension of information (Brooks, 2017 ; Kwon et al, 2018 ), the number of times participant had to stop their primary task to handle the robot (Brooks, 2017 ), trust in robot (De Visser and Parasuraman, 2011 ; Rosenthal et al, 2012 ; Hamacher et al, 2016 ), the participant's emotional state (Groom et al, 2010 ) and their influence on user impressions of the robot (Groom et al, 2010 ; Shiomi et al, 2013 ; Bajones et al, 2016 ; Engelhardt and Hansson, 2017 ; Kwon et al, 2018 ). Brooks ( 2017 ) devised a measurement scale of people's reaction to failure called the REACTION scale, which claims to compare different failure situations based on the severity of the failures, the context risk involved, and effectiveness of recovery strategy.…”

“…As such, many mobile robots use audio and speech to communicate robotic failures. Some use simple audio tones to gather user attention (e.g., Brooks, 2017 ), whereas others communicate failure using more complicated speech, such as Jibo 2 and the robot in Schütte et al ( 2017 ). Cha et al ( 2015 ) found that people perceived robots speaking conversationally as more capable than those that could only maintain a functional level of speech.…”

Referring to the recently seen: reference and perceptual memory in situated dialog

“…For example, a pronominal reference signals that the intended referent has a high degree of salience within the hearer's current mental model of the discourse context. 1 Examples of such research spanning the decades include [58,38,45,32,20,39,48,13,64,63,34,26,15,62,52] 2 Herskovits [27] provides an excellent overview of the challenges posed by spatial language. Many computational models of spatial language are based on the spatial template concept proposed by Logan and Sadler [54]; see [19,40,10,37] for examples of spatial template based computational models of the semantics of topological prepositions, and [18,42,5] for computational models of projective prepositions.…”

Situated Interaction with a Smart Environment: Challenges and Opportunities

“…Relative and intrinsic reference systems have frequently been used in automatic systems for establishing spatial reference, addressing various associated challenges [33]. For instance, Moratz and Tenbrink [26] aimed to capture speakers' spontaneous strategies by computational modelling of overlapping acceptance areas for specific subsets of projective terms, using either relative or intrinsic interpretations.…”

“…While there have been many efforts to capture the notion of common ground in general [8] and in human-robot interaction settings [6], the computational management of situated spatial dialogue is still under-developed [15,34] and requires creative solutions for reference handling [23], including attempts to incorporate the human's gaze in the system's interpretation procedure [1], and strategies for handling errors [33]. One major challenge concerns the fundamental difference between human concepts represented by natural language, especially in the domain of space [2], and formal systems suited for computational purposes, e.g., spatial reasoning-even if based on qualitative rather than metric relations [25].…”