Towards Visual Dialogue for Human-Robot Interaction

Part, Jose L.; García, Daniel Hernández; Yu, Yanchao; Gunson, Nancie; Dondrup, Christian; Lemon, Oliver

doi:10.1145/3434074.3447278

Cited by 3 publications

(5 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This allows us to create conversational systems which can assist blind and partially sighted people to find objects and access descriptions of places and scenes [4]. We have also developed interactive robots with the same capabilities [48]. Our spin-out company Alana AI is developing a visual conversational system for blind and partially sighted people in collaboration with the RNIB (Royal National Institute for the Blind).…”

Section: Assistive Visual Conversational Systems For Blind and Partia...mentioning

confidence: 99%

“…Many of our systems are developed with HRI use cases as their ultimate goal. These systems involve multimodal information in interaction, for example considering user gestures and facial expressions as well as speech [44], and they are situated and embodied, meaning that visual and spatial information must be considered [48]. Some systems involve multiple humans, meaning that multi-agent social information and decision-making needs to be modelled.…”

Section: Human-robot Interaction (Hri)mentioning

confidence: 99%

“…The SPRING project currently works on socially intelligent HRI in a multi-user scenario, in the context of a hospital waiting room. Here we deal with visual dialogue about the room (where to sit, lost objects etc) [48] as well as multi-party conversations tracking the individual goals of different users (e.g. one needs a coffee, another wants to check in) within the interaction context.…”

Section: Human-robot Interaction (Hri)mentioning

confidence: 99%

“…book a flight, find a restaurant, get information. Alana is an open-domain social bot, also used in the MuMMER and SPRING projects [20,48].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Conversational AI for multi-agent communication in Natural Language

Lemon

2022

AIC

Self Cite

View full text Add to dashboard Cite

Research at the Interaction Lab focuses on human-agent communication using conversational Natural Language. The ultimate goal is to create systems where humans and AI agents (including embodied robots) can spontaneously form teams and coordinate shared tasks through the use of Natural Language conversation as a universal communication interface. This paper first introduces machine learning approaches to problems in conversational AI in general, where computational agents must coordinate with humans to solve tasks using conversational Natural Language. It also covers some of the practical systems developed in the Interaction Lab, ranging from speech interfaces on smart speakers to embodied robots interacting using visually grounded language. In several cases communication between multiple agents is addressed. The paper surveys the central research problems addressed here, the approaches developed, and our main results. Some key open research questions and directions are then discussed, leading towards a future vision of conversational, collaborative multi-agent systems.

show abstract

Section: Assistive Visual Conversational Systems For Blind and Partia...mentioning

confidence: 99%

Section: Human-robot Interaction (Hri)mentioning

confidence: 99%

Section: Human-robot Interaction (Hri)mentioning

confidence: 99%

“…book a flight, find a restaurant, get information. Alana is an open-domain social bot, also used in the MuMMER and SPRING projects [20,48].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Conversational AI for multi-agent communication in Natural Language

Lemon

2022

AIC

Self Cite

View full text Add to dashboard Cite

show abstract

“…The SPRING system retains the rule-based and News bots, supplemented with a number of new, domain-specific bots. Visual Task Bot handles visual dialogue within the conversation, converting the user's inferred intent and any entities associated with it to a goal message that is forwarded to the visual action server(Part et al, 2021). Reception Bot welcomes visitors, helps them check-in and answers FAQs on, e.g., catering facilities and schedules.…”

mentioning

confidence: 99%

A Visually-Aware Conversational Robot Receptionist

Gunson,

Hernandez Garcia,

Sieińska

et al. 2022

Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

Socially Assistive Robots (SARs) have the potential to play an increasingly important role in a variety of contexts including healthcare, but most existing systems have very limited interactive capabilities. We will demonstrate a robot receptionist that not only supports task-based and social dialogue via natural spoken conversation but is also capable of visually grounded dialogue; able to perceive and discuss the shared physical environment (e.g. helping users to locate personal belongings or objects of interest). Task-based dialogues include check-in, navigation and FAQs about facilities, alongside social features such as chit-chat, access to the latest news and a quiz game to play while waiting. We also show how visual context (objects and their spatial relations) can be combined with linguistic representations of dialogue context, to support visual dialogue and question answering. We will demonstrate the system on a humanoid ARI robot, which is being deployed in a hospital reception area.

show abstract