Toward Personalized Affect-Aware Socially Assistive Robot Tutors for Long-Term Interventions with Children with Autism

Shi, Zhonghao; Groechel, Thomas R.; Jain, Shomik; Chima, Kourtney; Rudovic, Ognjen; Matarić, Maja J.

doi:10.1145/3526111

Cited by 19 publications

(6 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the definition of cognitive-affective states may vary in different social contexts. Due to the independent and identically distributed (IID) assumption made by ML model training (Wang et al 2022), existing ML models struggle to generalize effectively and quickly to test data that are distributed differently from the training data, particularly in the context of SAR (Shi et al 2022).…”

Section: Multimodal User Understandingmentioning

confidence: 99%

How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Survey

Shi,

Landrum,

O'Connell

et al. 2024

AAAI-SS

View full text Add to dashboard Cite

Socially assistive robots (SARs) have shown great success in providing personalized cognitive-affective support for user populations with special needs such as older adults, children with autism spectrum disorder (ASD), and individuals with mental health challenges. The large body of work on SAR demonstrates its potential to provide at-home support that complements clinic-based interventions delivered by mental health professionals, making these interventions more effective and accessible. However, there are still several major technical challenges that hinder SAR-mediated interactions and interventions from reaching human-level social intelligence and efficacy. With the recent advances in large language models (LLMs), there is an increased potential for novel applications within the field of SAR that can significantly expand the current capabilities of SARs. However, incorporating LLMs introduces new risks and ethical concerns that have not yet been encountered, and must be carefully be addressed to safely deploy these more advanced systems. In this work, we aim to conduct a brief survey on the use of LLMs in SAR technologies, and discuss the potentials and risks of applying LLMs to the following three major technical challenges of SAR: 1) natural language dialog; 2) multimodal understanding; 3) LLMs as robot policies.

show abstract

Section: Multimodal User Understandingmentioning

confidence: 99%

How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Survey

Shi,

Landrum,

O'Connell

et al. 2024

AAAI-SS

View full text Add to dashboard Cite

show abstract

“…Contrary to generic models, personalized models are fine-tuned given the data of a single user or user segment. Accounting for such interindividual variability has been proven to dramatically improve prediction performance in various tasks within the PI domain, such as pain detection, engagement estimation, and stress prediction from ubiquitous devices data [69,89,95]. Given the increasing popularity of the personalization paradigm, in this study, we investigate whether personalization as a modeling choice can amplify performance disparities across different user segments in the data, given the existence of representation bias.…”

Section: Learning Biasmentioning

confidence: 99%

Uncovering Bias in Personal Informatics

Yfantidou¹,

Sermpezis²,

Vakali³

et al. 2023

Preprint

View full text Add to dashboard Cite

Personal informatics (PI) systems, powered by smartphones and wearables, enable people to lead healthier lifestyles by providing meaningful and actionable insights that break down barriers between users and their health information. Today, such systems are used by billions of users for monitoring not only physical activity and sleep but also vital signs and women's and heart health, among others. Despite their widespread usage, the processing of sensitive PI data may suffer from biases, which may entail practical and ethical implications. In this work, we present the first comprehensive empirical and analytical study of bias in PI systems, including biases in raw data and in the entire machine learning life cycle. We use the most detailed framework to date for exploring the different sources of bias and find that biases exist both in the data generation and the model learning and implementation streams. According to our results, the most affected minority groups are users with health issues, such as diabetes, joint issues, and hypertension, and female users, whose data biases are propagated or even amplified by learning models, while intersectional biases can also be observed. CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing; • Applied computing → Consumer health; • Computing methodologies → Artificial intelligence; • Social and professional topics → Codes of ethics.

show abstract

“…Human posture estimation has been an important tool in various areas of human-robot interaction such as sociallyassistive robots [54,56], pHRI [25,64], teleoperation [18,40,77], engagement analysis [26] human-aware decision making and planning [9], and situation awareness [17]. Posture estimation in pHRI has been used mainly as a tool to derive other metrics for HRI evaluation.…”

Section: Posture Estimation In Human-robot Interactionmentioning

confidence: 99%

Occlusion-Robust Multi-Sensory Posture Estimation in Physical Human-Robot Interaction

Yazdani¹,

Novin²,

Hermans³

2022

Preprint

View full text Add to dashboard Cite

and NVIDIA, USA 3D posture estimation is important in analyzing and improving ergonomics in physical human-robot interaction and reducing the risk of musculoskeletal disorders. Vision-based posture estimation approaches are prone to sensor and model errors, as well as occlusion, while posture estimation solely from the interacting robot's trajectory suffers from ambiguous solutions. To benefit from the advantages of both approaches and improve upon their drawbacks, we introduce a low-cost, non-intrusive, and occlusion-robust multi-sensory 3D postural estimation algorithm in physical human-robot interaction. We use 2D postures from OpenPose over a single camera, and the trajectory of the interacting robot while the human performs a task. We model the problem as a partially-observable dynamical system and we infer the 3D posture via a particle filter. We present our work in teleoperation, but it can be generalized to other applications of physical human-robot interaction. We show that our multi-sensory system resolves human kinematic redundancy better than posture estimation solely using OpenPose or posture estimation solely using the robot's trajectory. This will increase the accuracy of estimated postures compared to the gold-standard motion capture postures. Moreover, our approach also performs better than other single sensory methods when postural assessment using RULA assessment tool.

show abstract

Toward Personalized Affect-Aware Socially Assistive Robot Tutors for Long-Term Interventions with Children with Autism

Cited by 19 publications

References 48 publications

How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Survey

How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Survey

Uncovering Bias in Personal Informatics

Occlusion-Robust Multi-Sensory Posture Estimation in Physical Human-Robot Interaction

Contact Info

Product

Resources

About