Single-Trial Recognition of Video Gamer’s Expertise from Brain Haemodynamic and Facial Emotion Responses

Abstract

With an increase in consumer demand of video gaming entertainment, the game industry is exploring novel ways of game interaction such as providing direct interfaces between the game and the gamers’ cognitive or affective responses. In this work, gamer’s brain activity has been imaged using functional near infrared spectroscopy (fNIRS) whilst they watch video of a video game (League of Legends) they play. A video of the face of the participants is also recorded for each of a total of 15 trials where a trial is defined as watching a gameplay video. From the data collected, i.e., gamer’s fNIRS data in combination with emotional state estimation from gamer’s facial expressions, the expertise level of the gamers has been decoded per trial in a multi-modal framework comprising of unsupervised deep feature learning and classification by state-of-the-art models. The best tri-class classification accuracy is obtained using a cascade of random convolutional kernel transform (ROCKET) feature extraction method and deep classifier at 91.44%. This is the first work that aims at decoding expertise level of gamers using non-restrictive and portable technologies for brain imaging, and emotional state recognition derived from gamers’ facial expressions. This work has profound implications for novel designs of future human interactions with video games and brain-controlled games.

Enhancing Therapeutic Processes in Videoconferencing Psychotherapy: Interview Study of Psychologists’ Technological Perspective

“…The field's high-level technology enabled video game players to share facial expressions, body language clues, and physiological indicators such as the heart beat in real time [70]. In a study conducted by Andreu-Perez et al [71], researchers were able to classify gamers' expertise by deciphering their emotions and brain information relying on their facial expressions. This means that advanced technology implemented in specific fields can possibly be adopted for multidisciplinary purposes, such as enhancing the VCP field.…”

Deep learning in fNIRS: a review

“…Andreu-Perez et al. 89 classified the expertise of subjects watching 30 s clips of the video game League of Legends using fNIRS data and facial expressions. The fully connected deep neural network (FCDNN) and deep classifier autoencoder (DCAE) were compared against many traditional machine learning techniques, including SVM and kNN in a three-class test to determine the skill level of the subject watching.…”

“…While most studies have focused on using cortical activations to classify when a specified task is being completed, a few studies have begun looking into predicting the skill level of the subject at a given task. Andreu-Perez et al 89 classified the expertise of subjects watching 30 s clips of the video game League of Legends using fNIRS data and facial expressions. The fully connected deep neural network (FCDNN) and deep classifier autoencoder (DCAE) were compared against many traditional machine learning techniques, including SVM and kNN in a threeclass test to determine the skill level of the subject watching.…”

“…While most studies have focused on using cortical activations to classify when a specified task is being completed, a few studies have begun looking into predicting the skill level of the subject at a given task. Andreu-Perez et al 89 . classified the expertise of subjects watching 30 s clips of the video game League of Legends using fNIRS data and facial expressions.…”

S-Rocket: Selective Random Convolution Kernels for Time Series Classification

“…Rocket [10] and MiniRocket [12] use random convolution kernels for transforming input time series into a set of features to train a linear classifier, without training the kernels. Both methods have shown fast and accurate time series classification on standard datasets and for different applications in a fraction of the training time of existing methods, such as UCR archive [15], inter-burst detection in electroencephalogram (EEG) signals [16], driver's distraction detection using EEG signals [17], functional near infrared spectroscopy signals classification [18], and human activity recognition [19].…”