Evaluation of Ten Open-Source Eye-Movement Classification Algorithms in Simulated Surgical Scenarios

Abstract

Despite providing several insights into visual attention and evidence regarding certain brain states and psychological functions, classifying eye movements is a highly demanding process. Currently, there are several algorithms to classify eye movement events which use different approaches. However, to date, only a limited number of studies have assessed these algorithms under specific conditions, such as those required for surgical training programmes. This study presents an investigation of ten open-source eye-movement classification algorithms using the Eye Tribe eye-tracker. The algorithms were tested on the eye-movement records obtained from 23 surgical residents, who performed computer-based surgical simulation tasks under different hand conditions. The aim was to offer data for the improvement of surgical training programmes. According to the results, due to the different classification methods and default threshold values, the ten algorithms produced different results. Considering the fixation duration, the only common event for all of the investigated algorithms, the binocular-individual threshold (BIT) algorithm resulted in a different clustering compared to the other algorithms. Based on the other set of common events, three clusters were determined by eight algorithms (except BIT and event detection (ED)), distinguishing dispersion-based, velocity-based and modified versions of velocity-based algorithms. Accordingly, it was concluded that dispersion-based and velocity-based algorithms provided different results. Additionally, as it individually specifies the threshold values for the eye-movement data, when there is no consensus about the threshold values to be set, the BIT algorithm can be selected. Especially for such cases like simulation-based surgical skill-training, the use of individualised threshold values in the BIT algorithm can be more beneficial in classifying the raw eye data and thus evaluating the individual progress levels of trainees based on their eye movement behaviours. In conclusion, the threshold values had a critical effect on the algorithm results. Since default values may not always be suitable for the unique features of different data sets, guidelines should be developed to indicate how the threshold values are set for each algorithm. INDEX TERMS Eye-movement classification algorithms, eye-movement events, eye-tracking.

RETRACTED ARTICLE: Eye tracking: empirical foundations for a minimal reporting guideline

“…The many existing eventdetection algorithms do not necessarily produce the same output measures when given the same eye-tracking data. In fact, several algorithm comparisons have reported large differences in fixation and saccade measures between algorithms (Andersson et al, 2017;Benjamins et al, 2018;Dalveren & Cagiltay, 2019;Komogortsev et al, 2010;Salvucci & Goldberg, 2000;Stuart et al, 2019). This research suggests that differences in, for instance, average fixation durations between studies that use different algorithms may in part stem for differences between the algorithms.…”

“…Attrition rate exhibits a large variation between studies. For instance, Dalveren and Cagiltay (2019) report an attrition rate of 17.9% for the EyeTribe, while Holmqvist (2015) report 1.0% for the same eye tracker. The reported attrition rates appear to be lower in studies with adult participants in light-controlled labs, for instance 0-8.2% in Holmqvist (2015), compared to recordings made in sun-lit environments, for instance Wang et al (2010), who report 32% attrition rate during outdoor driving.…”

REMoDNaV: robust eye-movement classification for dynamic stimulation

“…The former rely on velocity thresholds to differentiate between different eye-movement events, while the latter classify eye movements based on the size of the region the recorded data falls into for a given amount of time (Holmqvist et al, 2011 ). Both types of algorithms are common (see e.g., Hessels et al, ( 2017 ) for a recent dispersion-based, and e.g., van Renswoude et al, ( 2018 ) for a recent velocity-based solution, and see Dalveren and Cagiltay ( 2019 ) for an evaluation of common algorithms of both types). Like NH, REMoDNaV is a velocity-based event classification algorithm.…”

REMoDNaV: Robust Eye-Movement Classification for Dynamic Stimulation

“…The former rely on velocity thresholds to differentiate between different eye movement events, while the latter classify eye movements based on the size of the region the recorded data falls into for a given amount of time (Holmqvist et al, 2011). Both types of algorithms are common (see e.g., Hessels et al (2017) for a recent dispersion-based, and e.g., van Renswoude et al (2018) for a recent velocity-based solution, and see Dalveren and Cagiltay (2019) for an evaluation of common algorithms of both types). Like NH, REMoDNaV is a velocity-based event classification algorithm.…”