Toward Personalized Affect-Aware Socially Assistive Robot Tutors in Long-Term Interventions for Children with Autism

Abstract

Affect-aware socially assistive robotics (SAR) has shown great potential for augmenting interventions for children with autism spectrum disorders (ASD). However, current SAR cannot yet perceive the unique and diverse set of atypical cognitive-affective behaviors from children with ASD in an automatic and personalized fashion in long-term (multi-session) real-world interactions. To bridge this gap, this work designed and validated personalized models of arousal and valence for children with ASD using a multi-session in-home dataset of SAR interventions. By training machine learning (ML) algorithms with supervised domain adaptation (s-DA), the personalized models were able to trade off between the limited individual data and the more abundant less personal data pooled from other study participants. We evaluated the effects of personalization on a long-term multimodal dataset consisting of 4 children with ASD with a total of 19 sessions, and derived inter-rater reliability (IR) scores for binary arousal (IR = 83%) and valence (IR = 81%) labels between human annotators. Our results show that personalized Gradient Boosted Decision Trees (XGBoost) models with s-DA outperformed two non-personalized individualized and generic model baselines not only on the weighted average of all sessions, but also statistically (𝑝 < .05) across individual sessions. This work paves the way for the development of personalized autonomous SAR systems tailored toward individuals with atypical cognitive-affective and socio-emotional needs. CCS Concepts: • Human-centered computing → Empirical studies in HCI; • Applied computing → Computer-assisted instruction; • Computer systems organization → Robotics.

Real-time Feedback based on Emotion Recognition for Improving Children’s Metacognitive Monitoring Skill

“…1) showed that the transfer learning algorithm can improve the accuracy of facial emotional classification based on a generic classifier and a personalized classifier (see (a), (b) in Fig. 1 that the mean accuracy in middle is higher than both end), and this result is consistent with the experiment in [22,23,26]. Now we are requesting the permission of accessing an emotional dataset of children with ASD to test the accuracy of emotional classifier on children with ASD.…”

“…However, the relationship between emotion and metacognitive monitoring accuracy of children with ASD is still emerging. The Facial Emotion Expression (FEE) of children with ASD can not be recognised with a reliable accuracy [3,22,26,31], because children with ASD have unique and impaired emotion expressions [2,28]. Thus our research project aims to fill the following gaps:…”

“…(3) It needs to be explored which patterns of emotional expression are significantly different in children with ASD [26].…”

A game-based approach for evaluating and customizing handwriting training using an autonomous social robot

“…Algorithms that take more time to train their model and, consequently, present more computational costs are also used to address this issue. Machine learning algorithms with supervised domain adaptation (s-DA) to afford personalized models are examples of this kind of knowledge base for decision-making [18]. Evaluations of such methods on the effects of personalization on a long-term multimodal dataset showed that their outcomes outperformed non-personalized, individualized and generic model baselines in both individual sessions and also in the average of all sessions.…”

Personalized Affect-Aware Socially Assistive Robot Tutors Aimed at Fostering Social Grit in Children with Autism

“…As shown in Figure 1(B), we plan to develop the SAR tutor with a touchscreen tablet that will serve as a shared collaborative space for the child and robot. Each participant will take part in a first pilot session, whose multimodal data will be annotated and used to pre-train personalized affective models of arousal and valence using supervised domain adaptation we have already validated in our previous work [26]. Using real-time predictions of arousal and valence, we plan to apply an affective reinforcement learning approach in order to personalize the robot's responses to keep the child engaged in the ACT interventions.…”