A novel evaluation of two related and two independent algorithms for eye movement classification during reading

Abstract

Nystrӧm and Holmqvist have published a method for the classification of eye movements during reading (ONH) (Nyström & Holmqvist, 2010). When we applied this algorithm to our data, the results were not satisfactory, so we modified the algorithm (now the MNH) to better classify our data. The changes included: (1) reducing the amount of signal filtering, (2) excluding a new type of noise, (3) removing several adaptive thresholds and replacing them with fixed thresholds, (4) changing the way that the start and end of each saccade was determined, (5) employing a new algorithm for detecting PSOs, and (6) allowing a fixation period to either begin or end with noise. A new method for the evaluation of classification algorithms is presented. It was designed to provide comprehensive feedback to an algorithm developer, in a time-efficient manner, about the types and numbers of classification errors that an algorithm produces. This evaluation was conducted by three expert raters independently, across 20 randomly chosen recordings, each classified by both algorithms. The MNH made many fewer errors in determining when saccades start and end, and it also detected some fixations and saccades that the ONH did not. The MNH fails to detect very small saccades. We also evaluated two additional algorithms: the EyeLink Parser and a more current, machine-learning-based algorithm. The EyeLink Parser tended to find more saccades that ended too early than did the other methods, and we found numerous problems with the output of the machine-learning-based algorithm.

Why Temporal Persistence of Biometric Features, as Assessed by the Intraclass Correlation Coefficient, Is So Valuable for Classification Performance

“…Session 1 to Session 2 (task-to-task) time intervals ranged from 13 min to 42 min (mean = 19.5; SD = 4.2). For eye movement classification, we employed the MNH method described in [ 27 ]. It identifies fixation periods, saccades and post-saccadic oscillations (PSOs) as well as periods of artifact and noise.…”

REMoDNaV: Robust Eye-Movement Classification for Dynamic Stimulation

“…First, implausible spikes in the coordinate time series are removed with a heuristic spike filter (Stampe, 1993) (Figure 1, P1). This filter is standard in many eye tracking toolboxes and often used for preprocessing (e.g., Friedman et al, 2018). Data samples around signal loss (e.g., eye blinks) can be set to non-numeric values (NaN) in order to eliminate spurious movement signals without shortening the time series (dilate nan, min blink duration; Figure 1, P2).…”

“…REMoDNaV alters this algorithm by using robust statistics that are more suitable for the nonnormal distribution of velocities (Friedman et al, 2018), such that the new threshold is computed by:…”

“…And indeed, when we evaluated an algorithm by Nyström and Holmqvist (2010), one of the winners in the aforementioned comparison, on data from prolonged stimulation (≈15 min) with a feature film, we found the average and median durations of labeled fixations to exceed literature reports (e.g., Holmqvist et al, 2011;Dorr et al, 2010) by up to a factor of two. Additionally, and in particular for increasing levels of noise in the data, the algorithm clas-sified too few fixations, as also noted by Friedman et al (2018), because it discarded potential fixation events that contained data artifacts such as signalloss and distortion associated with blinks.…”

Correction to: “Using machine learning to detect events in eye-tracking data”

“…Finally, it has come to our attention (Friedman, Rigas, Abdulin, & Komogortsev, 2018) that on unseen data (not belonging to the training or validation set when the identification by random forest [IRF] algorithm was constructed), some of the events output by the IRF event detector were erroneously not removed or reclassified as other events, despite the fact that they violated the heuristic post-processing rules listed in the Bpost-processing^section of the article. This may have led to a slight The online version of the original article can be found at https://doi.org/ 10.3758/s13428-017-0860-3 * Raimondas Zemblys r.zemblys@tf.su.lt increase in the number of erroneous events included in the evaluation of IRF's performance reported in Zemblys et al (2018), dragging down IRF's performance slightly.…”