Gated Multimodal Fusion with Contrastive Learning for Turn-Taking Prediction in Human-Robot Dialogue

Yang, Jianguo; Peiying, Wang,; Feng, Mingchao; Chen, Meng; He, Xiaodong

doi:10.1109/icassp43922.2022.9747056

Cited by 8 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While gaze aversion and light indication helped decrease the interruptions, but they still occurred in most of the conversations. Turn-taking behavior can be further improved by using dialogue context (e.g., [75]), speech prosody (e.g., [76]), estimating the user's gaze, gestures, and facial expressions, such as eyebrow movement and mouth opening [77], or a combination of these features, e.g., [78][79][80] (for an in-depth review of turn-taking in HRI and conversational systems, see [81]) to prevent interrupting the user, both for improving user experience and speech recognition.…”

Section: Turn-takingmentioning

confidence: 99%

Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

Irfan

Kuoppamäki

Skantze

2023

Preprint

View full text Add to dashboard Cite

This work aims to provide initial guidelines towards developing companion robots with large language models (LLMs) to be part of everyday lives of older adults. Using iterative participatory design (co-design) approaches, we analyze the challenges of applying LLMs for multi-modal open-domain dialogue, deriving from older adults' (one-to-one) interactions with a personalized companion robot, built on Furhat robot with GPT-3.5. An initial study with 6 Swedish-speaking older adults (65 and older) showed that the robot frequently interrupted the users, responded slowly and repetitively, engaged in superficial conversations, and caused a barrier in the interaction due to foreign language (English). Upon incremental technical developments to address these issues, participatory design workshops were conducted with 28 Swedish-speaking older adults. While the interactions (in Swedish) were smoother, less disrupted, and more varied in topics and responses, further challenges were observed due to hallucinations and obsolete information, and disengagement cues, causing frustration, confusion, and worry.

show abstract

Section: Turn-takingmentioning

confidence: 99%

Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

Irfan

Kuoppamäki

Skantze

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Fine-grained address entity recognition from calls is an important task in many applications (Wu and Juang, 2022;Yang et al, 2022), such as obtaining delivery addresses from E-commerce assistant or post-sale service (Eligüzel et al, 2020). To be specific, the scenario considered in this paper is the extraction of fine-grained address entities distributed through the multi-turn spoken dialogue contexts.…”

Section: Introductionmentioning

confidence: 99%

Log-FGAER: Logic-Guided Fine-Grained Address Entity Recognition from Multi-Turn Spoken Dialogue

Han,

Wang,

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Fine-grained address entity recognition (FGAER) from multi-turn spoken dialogues is particularly challenging. The major reason lies in that a full address is often formed through a conversation process. Different parts of an address are distributed through multiple turns of a dialogue with spoken noises. It is nontrivial to extract by turn and combine them. This challenge has not been well emphasized by main-stream entity extraction algorithms. To address this issue, we propose in this paper a logic-guided fine-grained address recognition method (Log-FGAER), where we formulate the address hierarchy relationship as the logic rule and softly apply it in a probabilistic manner to improve the accuracy of FGAER. In addition, we provide an ontology-based data augmentation methodology that employs ChatGPT to augment a spoken dialogue dataset with labeled address entities. Experiments are conducted using datasets generated by the proposed data augmentation technique and derived from real-world scenarios. The results of the experiment demonstrate the efficacy of our proposal.

show abstract

Turn-Taking and Backchannel Prediction with Acoustic and Large Language Model Fusion

Wang,

Chen,

Khare

et al. 2024

ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Gated Multimodal Fusion with Contrastive Learning for Turn-Taking Prediction in Human-Robot Dialogue

Cited by 8 publications

References 18 publications

Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

Between Reality and Delusion: Challenges of Applying Large Language Models to Companion Robots for Open-Domain Dialogues with Older Adults

Log-FGAER: Logic-Guided Fine-Grained Address Entity Recognition from Multi-Turn Spoken Dialogue

Turn-Taking and Backchannel Prediction with Acoustic and Large Language Model Fusion

Contact Info

Product

Resources

About