Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

Irfan, Bahar; Kuoppamäki, Sanna-Mari; Skantze, Gabriel

doi:10.21203/rs.3.rs-2884789/v1

Cited by 8 publications

(22 citation statements)

References 277 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An example for such collaboration is a story-telling game CreativeBot [8] where a child and a robot Furhat create a story via turn-taking. On the other hand, the authors in [9] examine the difficulties associated with utilizing large language models for multi-modal open-domain dialogues, which derive from the interactions between older adults and a personalized companion robotthe Furhat robot, powered by GPT-3.5. Authors in [7] use the Pepper robot for chitchatting and enhance embodied dialogs by providing additional 'cloud-background' dialogue capabilities to complement the preexisting natural language understanding ability.…”

Section: Related Workmentioning

confidence: 99%

System software architecture for advancing human robot interaction by cloud services and multi-robot cooperation

Lekova,

Tsvetkova,

Andreeva

et al. 2024

IJITS JOURNAL

View full text Add to dashboard Cite

Human-like interactions with robots based on Conversational AI facilitate assistance and teamwork in various contexts. Those interactions are further enhanced by utilizing physical presence and context from the robot's hardware. Robot cooperation is also especially useful, when software or hardware resources have to be shared in a multi-robot system. Therefore, we propose a modular software architecture for multi-robot cooperation that extends the integration of Conversational AI into Socially Assistive Robots, previously suggested by authors. It utilizes a flow-based approach that involves shared repositories and direct or message-driven communication to convey natural language transcriptions among robots in order to support their cooperation. By experiments we evaluated the cooperation between NAOqi based robots and Furhat robot. Our experimental results demonstrate architecture's modularity and adaptability to different cloud services, along with its effectiveness for interactions involving multiple robots.

show abstract

Section: Related Workmentioning

confidence: 99%

System software architecture for advancing human robot interaction by cloud services and multi-robot cooperation

Lekova,

Tsvetkova,

Andreeva

et al. 2024

IJITS JOURNAL

View full text Add to dashboard Cite

show abstract

“…Our prior work (Irfan et al, 2023) (among others described in Section 2.2) provides a starting point for using a 10.3389/frobt.2024.1363713 foundation model (e.g., LLM) for a conversational companion robot for older adults. Deriving from the expectations of older adults outlined in the previous sections, and the challenges of LLMs encountered in our prior work based on the interactions with older adults, we offer actionable design recommendations for developing conversational companion robots that leverage foundation models, such as LLMs, vision-language models, and state-of-the-art architectures as their core, with potential relevance for other conversational robots and agents.…”

Section: Design Recommendationsmentioning

confidence: 99%

“…The generation of text or responses that seem plausible but factually incorrect is referred to as "hallucination" in foundation models, which is a commonly recognized challenge (Weidinger et al, 2021;Irfan et al, 2023). Attention mechanisms, regularization techniques, retrieval-based methods, evaluating…”

Section: Information Credibility and Recencymentioning

confidence: 99%

“…Empathy in dialogue can be conveyed through appreciation, agreement, and sharing of personal experiences (Lee et al, 2022), which can be achieved in LLMs that are shown to have high emotional awareness (Elyoseph et al, 2023). Prompting the model to be empathetic helps tailor its responses accordingly (e.g., Chen S. et al, 2023;Irfan et al, 2023). In addition, LLMs can be combined with supervised emotion recognition architectures (e.g.…”

Section: Reflection Of Congruent Emotionsmentioning

confidence: 99%

See 1 more Smart Citation

Recommendations for designing conversational companion robots with older adults through foundation models

Irfan,

Kuoppamäki,

Skantze

2024

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

Companion robots are aimed to mitigate loneliness and social isolation among older adults by providing social and emotional support in their everyday lives. However, older adults’ expectations of conversational companionship might substantially differ from what current technologies can achieve, as well as from other age groups like young adults. Thus, it is crucial to involve older adults in the development of conversational companion robots to ensure that these devices align with their unique expectations and experiences. The recent advancement in foundation models, such as large language models, has taken a significant stride toward fulfilling those expectations, in contrast to the prior literature that relied on humans controlling robots (i.e., Wizard of Oz) or limited rule-based architectures that are not feasible to apply in the daily lives of older adults. Consequently, we conducted a participatory design (co-design) study with 28 older adults, demonstrating a companion robot using a large language model (LLM), and design scenarios that represent situations from everyday life. The thematic analysis of the discussions around these scenarios shows that older adults expect a conversational companion robot to engage in conversation actively in isolation and passively in social settings, remember previous conversations and personalize, protect privacy and provide control over learned data, give information and daily reminders, foster social skills and connections, and express empathy and emotions. Based on these findings, this article provides actionable recommendations for designing conversational companion robots for older adults with foundation models, such as LLMs and vision-language models, which can also be applied to conversational robots in other domains.

show abstract

“…LLMs have opened up new possibilities in the field of robotics and human-robot teaming, most apparently for social robots and Socially Assistive Robots (SARs) (Alessa and Al-Khalifa, 2023;Irfan et al, 2023;Kahambing, 2023;Lee et al, 2023;Lekova et al, 2023). However, the conversational capabilities of LLMs extend beyond mere social interactions; their proficiency in handling a diverse range of textual inputs-without the need for rigidly predefined formats-marks a significant advancement for applications such as speech-controlled robotics, which have historically faced challenges with processing unstructured input.…”

Section: Llms and Robotsmentioning

confidence: 99%

Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation

Lakhnati,

Pascher,

Gerken

2024

Front. Robot. AI

View full text Add to dashboard Cite

In a rapidly evolving digital landscape autonomous tools and robots are becoming commonplace. Recognizing the significance of this development, this paper explores the integration of Large Language Models (LLMs) like Generative pre-trained transformer (GPT) into human-robot teaming environments to facilitate variable autonomy through the means of verbal human-robot communication. In this paper, we introduce a novel simulation framework for such a GPT-powered multi-robot testbed environment, based on a Unity Virtual Reality (VR) setting. This system allows users to interact with simulated robot agents through natural language, each powered by individual GPT cores. By means of OpenAI’s function calling, we bridge the gap between unstructured natural language input and structured robot actions. A user study with 12 participants explores the effectiveness of GPT-4 and, more importantly, user strategies when being given the opportunity to converse in natural language within a simulated multi-robot environment. Our findings suggest that users may have preconceived expectations on how to converse with robots and seldom try to explore the actual language and cognitive capabilities of their simulated robot collaborators. Still, those users who did explore were able to benefit from a much more natural flow of communication and human-like back-and-forth. We provide a set of lessons learned for future research and technical implementations of similar systems.

show abstract

Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

Cited by 8 publications

References 277 publications

System software architecture for advancing human robot interaction by cloud services and multi-robot cooperation

System software architecture for advancing human robot interaction by cloud services and multi-robot cooperation

Recommendations for designing conversational companion robots with older adults through foundation models

Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation

Contact Info

Product

Resources

About