Automated Coach to Practice Conversations

Abstract

Abstract-We present a real-time system including a 3D character that can converse, capture, analyze and interpret subtle and multidimensional human nonverbal behaviors for possible applications such as job interviews, public speaking, or even automated speech therapy. The system works in a personal computer and senses nonverbal data from video (i.e., facial expressions) and audio (i.e., speech recognition and prosody analysis) using a standard webcam. We contextualized the development and evaluation of our system as a training scenario for job interviews. Using user-centered design and iterations, we determine how the nonverbal data could be presented to the user in an intuitive and educational manner. We tested efficacy of the system in the context of job interviews with 90 MIT undergraduate students. Our results suggest that the participants who used our system to improve their interview skills were perceived to be better candidates by human judges. Participants reported that the most useful feature was being given feedback on their speaking rate, and overall they reported strong agreement that they would consider using this system again for self-reflection. Virtual agent; nonverbal behaviors; feedback; prosody; facial expressions.

Communication Skills Training Intervention Based on Automated Recognition of Nonverbal Signals

“…Researchers found that feedback of social signals during job interview training signifcantly improved in comparison to traditional methods of training. Similar results were found when attempting to improve communication skills in those with social impairments [60], public speaking [13,52,53], medical students [39], job interviews [3,9,31,32,[41][42][43]. However, no studies investigate whether social skills can be improved using automated feedback in the context of media skills training.…”

“…Studies using technology enhanced training have been successful in improving social skills or communication skills [31]. Researchers [27] developed My Automated Conversation CoacH (MACH).…”

“…Most training interventions rely on human trainers to provide feedback to learners, but this is costly, labor intensive, subjective and heavily reliant on the skill and experience of individual trainers. Research within the domain of afective computing and social signals processing have started to explore the potential of augmenting or replacing human trainers through the use of automated recognition of nonverbal signals with promising results [7,14,30,31,59]. However, most studies have focused on a narrow range of channels.…”