Affective Physiological State Analysis and Interpretable Predictive Modeling of Marksmanship in Go/NoGo VR Shooting Difficulty Task

Abstract

<p>This work has been submitted to the IEEE Transactions on Affective Computing for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.</p> <p><br></p> <p>Overburdening an individual’s limited cognitive resources, especially when engaged in critical operations, may result in disastrous mishaps. Regular assessments of individuals’ physiological states and associated performance become vital to improving their mission readiness in such scenarios. As a key step towards a field-ready system, this treatise discusses the experimental findings pertinent to affective physiological state modulation and predictive modeling of marksmanship during a Go/NoGo shooting task in an immersive virtual reality environment. The shooting exercise requires the participants to hit the enemy targets and spare the friendly targets. The shooting difficulty levels (SDLs) are introduced by modulating the subject-specific target exposure time. The physiological signals used for analysis comprise electrocardiogram (ECG), 64-channel electroencephalogram (EEG), and standard shooting performance scores from 31 subjects. Experimental results with ECG features encompass involuntary physiologic process regulation and the interplay between the autonomic nervous system (ANS) components varying with SDL. Similarly, EEG features highlight the variations in brain region activations with SDLs. Predictive modeling of shooting performance (enemy hit, friendly spare, overall score) and behavioral response (mean enemy reaction time) from physiological (ECG and EEG) features evince the potency of physiological sensing for marksmanship estimation in operational contexts. Moreover, interpretable Shapley value analysis of the predictive models comprehend the (positive/negative) marginal impact of the underlying physiological features on marksmanship. This multimodal physiological sensing framework may assess the alterations in psychophysiological affective states and cognitive effects for performance analysis in operational contexts.</p>