Assessing expertise using eye tracking in a Virtual Reality flight simulation

Abstract

Objective: The aim of this work was to examine the fidelity and validity of an aviation simulation using eye tracking. Background: Commercial head-mounted virtual reality (VR) systems offer a convenient and cost-effective alternative to existing aviation simulation (e.g., for refresher exercises). We performed pre-implementation testing of a novel aviation simulation, designed for head-mounted VR, to determine its fidelity and validity as a training device. Method: Eighteen airline pilots, with varying levels of flight experience, completed a sequence of training ‘flows’. Self-reported measures of presence and workload and users’ perceptions of fidelity were taken. Pilots’ eye movements and performance were recorded to determine whether more experienced pilots showed distinct performance and eye gaze profiles in the simulation, as they would in the real-world. Results: Real-world expertise correlated with eye gaze patterns characterised by fewer, but longer, fixations and a scan path that was more structured and less random. Multidimensional scaling analyses also indicated differential clustering of strategies in more versus less experienced pilots. Subjective ratings of performance, however, showed little relationship with real-world expertise or eye movements. Conclusion: We adopted an evidence-based approach to assessing the fidelity and validity of a VR flight training tool. Pilot reports indicated the simulation was realistic and potentially useful for training, while direct measurement of eye movements was useful for establishing construct validity and psychological fidelity of the simulation.

Exploring the role of virtual reality in military decision training

“…Questions such as "I had a sense of being there in the training environment" are responded to on a 1-7 Likert scale anchored between "at no time" and "almost all the time". This questionnaire has previously been validated in relation to experimentally induced breaks in presence (Slater and Steed, 2000), and used in the context of medical assessment scenarios (Pan et al, 2016), aviation simulation (Harris et al, 2022), and social interaction (Pan et al, 2015). The presence questionnaire was used for the VR and 2D video conditions as the questions are not applicable to real-world activities.…”