Multimodal User Feedback During Adaptive Robot-Human Presentations

Abstract

Feedback is an essential part of all communication, and agents communicating with humans must be able to both give and receive feedback in order to ensure mutual understanding. In this paper, we analyse multimodal feedback given by humans towards a robot that is presenting a piece of art in a shared environment, similar to a museum setting. The data analysed contains both video and audio recordings of 28 participants, and the data has been richly annotated both in terms of multimodal cues (speech, gaze, head gestures, facial expressions, and body pose), as well as the polarity of any feedback (negative, positive, or neutral). We train statistical and machine learning models on the dataset, and find that random forest models and multinomial regression models perform well on predicting the polarity of the participants' reactions. An analysis of the different modalities shows that most information is found in the participants' speech and head gestures, while much less information is found in their facial expressions, body pose and gaze. An analysis of the timing of the feedback shows that most feedback is given when the robot makes pauses (and thereby invites feedback), but that the more exact timing of the feedback does not affect its meaning.

Working with troubles and failures in conversation between humans and robots: workshop report

“…The robot presenter uses a knowledge graph to model the knowledge it is about to present, and then uses that same graph to keep track of the "grounding status" of the different pieces of information (Axelsson and Skantze, 2020). Multimodal feedback from the user (e.g., gaze, facial expressions, nods and backchannels) are interpreted as negative or positive, and the graph is updated accordingly, so that the presentation can be adapted to the user's level of knowledge and understanding (Axelsson and Skantze, 2022).…”

Modeling Feedback in Interaction With Conversational Agents—A Review

“…Recently, Axelsson and Skantze (2022) showed that individuals interacting with a system presenting a painting to them generally used feedback in the head and speech modalities, and that feedback in the facial, body pose and gaze modalities was not important for classifying their response as positive, negative, or neutral. These results give an indication that facial expressions are not important to sense for a presenting system of this kind, but may not be applicable in scenarios where the audience is more involved with the interaction than being the audience.…”

Verbal and nonverbal feedback signals in response to increasing levels of miscommunication

“…When listeners have difficulty hearing or understanding their interlocutor, they typically use a set of conversational strategies [1] to request a repetition [2], correction [3], or clarification of what was said [4,5]. These repair initiators can take different linguistic forms, ranging from short non-lexical sounds ("hmm?")…”