Multimodal Observation and Classification of People Engaged in Problem Solving: Application to Chess Players

Abstract

In this paper we present the first results of a pilot experiment in the interpretation of multimodal observations of human experts engaged in solving challenging chess problems. Our goal is to investigate the extent to which observations of eye-gaze, posture, emotion and other physiological signals can be used to model the cognitive state of subjects, and to explore the integration of multiple sensor modalities to improve the reliability of detection of human displays of awareness and emotion. Domains of application for such cognitive model based systems are, for instance, healthy autonomous ageing or automated training systems. Abilities to observe cognitive abilities and emotional reactions can allow artificial systems to provide appropriate assistance in such contexts. We observed chess players engaged in problems of increasing difficulty while recording their behavior. Such recordings can be used to estimate a participant's awareness of the current situation and to predict ability to respond effectively to challenging situations. Feature selection has been performed to construct a multimodal classifier relying on the most relevant features from each modality. Initial results indicate that eye-gaze, body posture and emotion are good features to capture such awareness. This experiment also validates the use of our equipment as a general and reproducible tool for the study of participants engaged in screen-based interaction and/or problem solving.

The Role of Emotion in Problem Solving: First Results from Observing Chess

“…The tasks were presented with the subject alternately playing black or white. Full results with eye-gaze and fixation are discussed in [13]. Here we concentrate on observations of emotion.…”

“…As a pilot study, we observed expert chess players engaged in solving problems of increasing difficulty [13]. During the study, data were recorded from all sensors (Kinect 2, Webcam, Screen capture, user clicks, Tobii-Bar) using the RGBD Sync SDK from the Mobile RGBD project.…”

Deep learning investigation for chess player attention prediction using eye-tracking and game data

“…The reported results may hide that there are still rooms for improvement, the underlying bottleneck of these researches on chess remaining the amount of available data. The available recordings of eye tracking data in chess context [Guntz et al, 2018] represent a huge amount of work and hours of data. Nevertheless, they contain few configurations and around 30 players recorded in actual chess problem solving tasks.…”

“…This data was acquired for an experimentation on observing and interpreting subjects in problem solving tasks (chess in particular) through multimodal observations (eye tracking, posture, facial emotion, etc.) [Guntz et al, 2018].…”

“…The initial dataset comes from a multimodal study with people solving chess problems [Guntz et al, 2018]. For these experiments, they recorded the eye-gaze of 30 players (10-51 years, 2 female, age: M = 30.0; SD = 12.9) from various levels (from beginners to experts, with a majority of intermediate players), trying to solve 13 chess tasks of increasing difficulty in limited time:…”