Embodied conversational agents for multimodal automated social skills training in people with autism spectrum disorders

Abstract

Social skills training, performed by human trainers, is a well-established method for obtaining appropriate skills in social interaction. Previous work automated the process of social skills training by developing a dialogue system that teaches social communication skills through interaction with a computer avatar. Even though previous work that simulated social skills training only considered acoustic and linguistic information, human social skills trainers take into account visual and other non-verbal features. In this paper, we create and evaluate a social skills training system that closes this gap by considering the audiovisual features of the smiling ratio and the head pose (yaw and pitch). In addition, the previous system was only tested with graduate students; in this paper, we applied our system to children or young adults with autism spectrum disorders. For our experimental evaluation, we recruited 18 members from the general population and 10 people with autism spectrum disorders and gave them our proposed multimodal system to use. An experienced human social skills trainer rated the social skills of the users. We evaluated the system’s effectiveness by comparing pre- and post-training scores and identified significant improvement in their social skills using our proposed multimodal system. Computer-based social skills training is useful for people who experience social difficulties. Such a system can be used by teachers, therapists, and social skills trainers for rehabilitation and the supplemental use of human-based training anywhere and anytime.

Conversational agents in healthcare: a systematic review

“…Out of 14 conversational agents, 5 were embodied conversational agents 20 , 22–24 , 35 and 2 were chatbots; 21 , 27 the remaining were unspecific conversational agents. …”

“…Across the set of conversational agents we identified, task-oriented conversational agents were the most common (8 of 14, evaluated in 11 studies), where the goal was to assist a human user to complete a specific task 20 , 22 , 23 , 25 , 29–35 . Tasks included automating clinical diagnostic interviews, data collection, and telemonitoring.…”

“…The most common output was also spoken natural language (7 of 14, evaluated in 10 studies) 22 , 23 , 25 , 29–35 . The remainder either used written natural language (4 of 14, evaluated in 4 studies), 19 , 24 , 27 , 28 or a combination of written and spoken language (3 of 14, evaluated in 3 studies) 20 , 21 , 26 …”

“…A further 6 studies evaluated 3 different conversational agents used in applications supporting both clinicians and patients in telemonitoring and data collection 30–34 Table 2.Study characteristics and results from the evaluation of conversational agents supporting patients and consumersAuthor, year a Health domainCA purposeStudy type and methodsEvaluation measures and main findingsTechnical performanceUser experienceHealth-related measuresTechnology supporting patients and consumers b Fitzpatrick et al., 2017 19 Mental health (depression, anxiety)Psychotherapy support, education RCT [2-week trial; 70 participants with symptoms of depression and anxiety; group 1—CA delivering CBT, group 2—educational eBook]NR• High overall satisfaction (4.3/5 Likert scale)• Participants interacted with the CA 12.1 times• Issues in spoken language understanding• Reduced depression symptoms (PHQ-9): effect size d =0.44, p =.04• No change in anxiety symptoms (GAD-7) or affect (PANAS)Tanaka et al., 2017 20 Mental health (autism)Social skills practice, education Quasi-experimental [ Study 1 : 2 groups (group 1—feedback based on audio features, group 2—audiovisual feedback), 18 students; Study 2 : 1 group, 10 people with autism]• Narrative skills score: 1 (not good) to 7 (good) [scale NR]NRNRImproved narrative skills scores (pre-post, one-tailed):• Study 1 (audiovisual feedback): d =0.98; p =.03)• Study 2: d =1.17; p =.003Miner et al., 2016 26 Mental and physical health, violenceQuestion answering, personal assistance, conversational Cross-sectional [Smartphones’ CAs (Siri, Google Now, S Voice, and Cortana) were asked 9 questions; responses were analyzed according to the CA’s ability to 1) recognize a crisis, 2) respond with respectful language, 3) refer to an appropriate helpline or other health resources]NR• CAs frequently did not recognize the health concern• Responses were often incomplete and inconsistent• Referral to appropriate health resources was rare• No variation in responses by tone or sex of the user• Issues in spoken language understanding and/or dialogue management• Siri, Google Now, and S Voice responded appropriately to the statement “I want to commit suicide”; Siri and Google Now referred the user to a suicide prevention helpline• Siri recognized physical concerns and referred to nearby medical facilitiesIreland et al., 2016…”

“…Risk of bias assessment for the RCTs showed moderate to high risk (Supplementary Material S4); assessment of quasi-experimental studies was not possible due to the quality of reporting. Conflict of interest statements were missing from 10 studies, 21 , 23–25 , 27 , 29–31 , 33 , 34 6 reported no conflict of interest, 20 , 22 , 26 , 28 , 32 , 35 and 1 disclosed a relevant financial conflict of interest (Supplementary Material S5) 19 . In 3 studies, sources of funding were not reported 21 , 23 , 26 …”