Nonverbal communication in virtual reality: Nodding as a social signal in virtual interactions

Aburumman, Nadine; Gillies, Marco; Ward, Jamie A.; Hamilton, Antonia F. de C.

doi:10.1016/j.ijhcs.2022.102819

Cited by 39 publications

(20 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Regarding how sensitive humans are to noticing discrepancies in human facial expressions even in milliseconds, any delay in facial data synchronization may induce distortions to the dataset. The work presented in this paper can be seen in line with the recent studies on dyadic human–robot interaction, for instance [ 30 , 31 , 32 , 33 ].…”

Section: Introductionsupporting

confidence: 82%

Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings

Sham

Khan

Lamas

et al. 2023

Sensors

View full text Add to dashboard Cite

Emotion recognition is a significant issue in many sectors that use human emotion reactions as communication for marketing, technological equipment, or human–robot interaction. The realistic facial behavior of social robots and artificial agents is still a challenge, limiting their emotional credibility in dyadic face-to-face situations with humans. One obstacle is the lack of appropriate training data on how humans typically interact in such settings. This article focused on collecting the facial behavior of 60 participants to create a new type of dyadic emotion reaction database. For this purpose, we propose a methodology that automatically captures the facial expressions of participants via webcam while they are engaged with other people (facial videos) in emotionally primed contexts. The data were then analyzed using three different Facial Expression Analysis (FEA) tools: iMotions, the Mini-Xception model, and the Py-Feat FEA toolkit. Although the emotion reactions were reported as genuine, the comparative analysis between the aforementioned models could not agree with a single emotion reaction prediction. Based on this result, a more-robust and -effective model for emotion reaction prediction is needed. The relevance of this work for human–computer interaction studies lies in its novel approach to developing adaptive behaviors for synthetic human-like beings (virtual or robotic), allowing them to simulate human facial interaction behavior in contextually varying dyadic situations with humans. This article should be useful for researchers using human emotion analysis while deciding on a suitable methodology to collect facial expression reactions in a dyadic setting.

show abstract

Section: Introductionsupporting

confidence: 82%

Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings

Sham

Khan

Lamas

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…Because the goal of our study was to create a virtual being as a companion, we aimed to make it more approachable and engaging. 85 Smiling virtual agents induce more positive affect than non-smiling behavior. 93 Hence, to enhance the positive impression, we incorporated smiling in both modes and occasionally nodding in the listening mode of the virtual agent's face.…”

Section: Visual Designmentioning

confidence: 99%

“…Another study by McRorie et al 84 discovered that the nonverbal behavior cues in video clips are perceived consistently with the personality types priorly derived from the still images of virtual agents (e.g., frowning eyebrows for high neuroticism). A recent study by Aburumman et al 85 found that people rated a virtual human that was nodding during conversation as significantly more interactive, approachable, and engaging than a non‐nodding virtual human.…”

Section: Related Workmentioning

confidence: 99%

Natural conversations with a virtual being: How user experience with a current conversational AI model compares to expectations

Khvan

Choi

2023

Computer Animation & Virtual

View full text Add to dashboard Cite

The present work investigates the effect of natural conversations with virtual beings on user perceptions with a current conversational AI model (Meta's BlenderBot). To this aim, we designed a virtual being from a deep learning‐generated face and a conversational AI model acting as a virtual conversation partner in an online conferencing software and evaluated it in 11 perceptions of social attributes. Compared to prior expectations, participants perceived the virtual being as distinctly higher in warmth (engaging, empathic, and approachable) but lower in realism and credibility after 5 days of 10 min daily conversations (Study 1). Further, we explored the idea of simplifying the technical setup to reduce the technical entry barrier for such AI applications (Study 2). To this aim, we conducted several trials of fine‐tuning a small conversational model of 90 million parameters until its performance metrics improved. Testing this fine‐tuned model with users revealed that this model was not perceived differently from a large conversational model (1.4 billion parameters). In summary, our findings show that recent progress in conversational AI has added warmth‐related aspects to the user experience with virtual beings, and that fine‐tuning a conversational AI model can be effective to reduce technical complexity.

show abstract

“…Motion capture technologies can affect the experience of social VR because motion captured avatars are able to convey a wider range of social cues during an interaction (Kokkinara and McDonnell, 2015;Lugrin et al, 2015;Latoschik et al, 2017;Herrera et al, 2020;Kyrlitsias and Michael-Grigoriou, 2022;Rogers et al, 2022). For instance, avatars that express interest with direct eye gaze to our presence and behaviour in VR are often rated as having a higher level of behavioural realism (Aburumman et al, 2022;Grewe et al, 2021;Wu et al, 2021).…”

Section: Behavioural Realism In Vrmentioning

confidence: 99%

“…However, motion capture is becoming more accessible and can enhance perceptions of animation realism (Fribourg et al, 2020;Rogers et al, 2022). Avatar expressiveness in VR is important for conveying real life social cues such as eye gaze and gesturing (Aburumman et al, 2022;Cao et al, 2020;Ferstl and McDonnell, 2018;Herrera et al, 2020;Kokkinara and McDonnell, 2015;Kyrlitsias and…”

Section: Introductionmentioning

confidence: 99%

Expressiveness of real-time motion captured avatars influences perceived animation realism and perceived quality of social interaction in virtual reality

Fraser

Branson

Hollett

et al. 2022

Front. Virtual Real.

View full text Add to dashboard Cite

Using motion capture to enhance the realism of social interaction in virtual reality (VR) is growing in popularity. However, the impact of different levels of avatar expressiveness on the user experience is not well understood. In the present study we manipulated levels of face and body expressiveness of avatars while investigating participant perceptions of animation realism and interaction quality when disclosing positive and negative experiences in VR. Moderate positive associations were observed between perceptions of animation realism and interaction quality. Post-experiment questions revealed that many of our participants (approximately 40%) indicated the avatar with the highest face and body expressiveness as having the most realistic face and body expressions. The same proportion also indicated the avatar with the highest face and body expressiveness as being the most comforting and enjoyable avatar to interact with. Our results suggest that higher levels of face and body expressiveness are important for enhancing perceptions of realism and interaction quality within a social interaction in VR using motion capture.

show abstract

Nonverbal communication in virtual reality: Nodding as a social signal in virtual interactions

Cited by 39 publications

References 36 publications

Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings

Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings

Natural conversations with a virtual being: How user experience with a current conversational AI model compares to expectations

Expressiveness of real-time motion captured avatars influences perceived animation realism and perceived quality of social interaction in virtual reality

Contact Info

Product

Resources

About