Immersive machine learning for social attitude detection in virtual reality narrative games

Abstract

People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program Non-Player Characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We collaborated with two game studios to develop an immersive machine learning (ML) pipeline for detecting social engagement. We collected data from participants-NPC interaction in VR, which was then annotated in the same immersive environment. Game design is a creative process and it is vital to respect designer’s creative vision and judgement. We therefore view annotation as a key part of the creative process. We trained a reinforcement learning algorithm (PPO) with imitation learning rewards using raw data (e.g. head position) and socially meaningful derived data (e.g. proxemics); we compared different ML configurations including pre-training and a temporal memory (LSTM). The pre-training and LSTM configuration using derived data performed the best (84% F1-score, 83% accuracy). The models using raw data did not generalise. Overall, this work introduces an immersive ML pipeline for detecting social engagement and demonstrates how creatives could use ML and VR to expand their ability to design more engaging experiences. Given the pipeline’s results for social engagement detection, we generalise it for detecting human-defined social attitudes.

More than buttons on controllers

“…During the interaction, the VC carried out a monologue about her family and her life. The monologue was written during a writers room organised by a national centre for immersive storytelling (see Table 1 and the supplementary video in [Dobre et al 2022]).…”

“…The interaction from 𝑆3 aimed to produce primarily low social engagement data. For an example of the participants' behaviour in each part, see videos in supplementary material from [Dobre et al 2022].…”

“…The user's distance from the VC is calculated based on Hall's personal space [Hall 1966]. See [Dobre et al 2022] for more details on this feature. Both models (derived and raw data) use the human's annotations as ground truth data, and its output is a discrete binary value.…”

“…The use of psychologically inspired derived features is therefore a better approach. For a detailed discussion on the ML results, see [Dobre et al 2022].…”

“…All data supporting this study are provided as supplementary information accompanying [Dobre et al 2022] at DOI:10.1007/𝑠10055 − 022 − 00644 − 4.…”