Using Expressive Avatars to Increase Emotion Recognition: A Pilot Study

Comparing technologies for conveying emotions through realistic avatars in virtual reality‐based metaverse experiences

“…The second modality, later referred to as Lip‐Sync approximation (LS), did not rely on additional hardware modules, as it used real‐time algorithms to generate a plausible facial motion starting from an audio clip. To this purpose, similarly to what was done in Reference 12, the SALSA LipSync Suite asset was employed. Along with the main asset, the One‐Click automatic configurator for Adobe Character Generator was used to guarantee a smooth integration with the considered humanoid mesh, along with the Eyes module to generate a plausible random eye motion.…”

“…In order to better evaluate the contribution of the studied modalities in terms of emotion conveyance, it was decided to perform an analysis per single emotion, similarly to what done in Reference 12. The authors of that work focused on three main emotions (happiness, sadness, and anger).…”

“…The use of these techniques as an alternative to audio alone has been already studied. For instance, Hube et al, 12 extended a previous work 26 to examine the influence of facial visual parameters on virtual avatars in VR. Starting from a pre‐determined set of audio files, the authors extracted the emotions represented into them and operated a real‐time lip sync approximation (through SALSA Lip Sync Suite) with the aim to present this information on a virtual avatar in a VE.…”

“…Capturing facial expressions, however, is still challenging when using, for example, a Head‐Mounted Display (HMD) for VR, although devices supporting this functionality are appearing on the market given the importance of these social cues for emotions conveyance 11 . An alternative way to reproduce facial expressions in these situations is to use lip synchronization ( lip sync ), that is, synchronized animations on a speaking avatar 12 …”

“…11 An alternative way to reproduce facial expressions in these situations is to use lip synchronization (lip sync), that is, synchronized animations on a speaking avatar. 12 Moving from these premises, this work operates a comparison between the above modalities for conveying emotions through realistic full-body avatars in a multi-user VR-based scenario. The two modalities, implemented with currently available consumer-level hardware (HTC Vive Facial Tracker for the first one) and software (SALSA Lip Sync Suite v2 for the second one), have been evaluated by means of a user study involving 28 participants.…”