One algorithm to rule them all? An evaluation and discussion of ten eye movement event-detection algorithms

Abstract

Almost all eye-movement researchers use algorithms to parse raw data and detect distinct types of eye movement events, such as fixations, saccades, and pursuit, and then base their results on these. Surprisingly, these algorithms are rarely evaluated. We evaluated the classifications of ten eyemovement event detection algorithms, on data from an SMI HiSpeed 1250 system, and compared them to manual ratings of two human experts. The evaluation focused on fixations, saccades, and post-saccadic oscillations. The evaluation used both event duration parameters, and sample-by-sample comparisons to rank the algorithms. The resulting event durations varied substantially as a function of what algorithm was used. This evaluation differed from previous evaluations by considering a relatively large set of algorithms, multiple events, and data from both static and dynamic stimuli. The main conclusion is that current detectors of only fixations and saccades work reasonably well for static stimuli, but barely better than chance for dynamic stimuli. Differing results across evaluation methods make it difficult to select one winner for fixation detection. For saccade detection, however, the algorithm by Larsson, Nyström and Stridh (IEEE Transaction on Biomedical Engineering, 60(9): [2484][2485][2486][2487][2488][2489][2490][2491][2492][2493]2013) outperforms all algorithms in data from both static and dynamic stimuli. The data also show how improperly selected algorithms applied to dynamic data misestimate fixation and saccade properties.

The effect of sampling rate and lowpass filters on saccades – A modeling approach

“…It would be interesting to see which filters are most suitable for the different methods and what amount of left-over noise or introduced ringing is tolerable. This could be a future application of our saccade model, further extending the finding by Andersen, Larsson, Holmqvist, Stridh, and Nystöm (2016).…”

Is human classification by experienced untrained observers a gold standard in fixation detection?

“…Although Andersson et al (2017) suggest that we can agree in simple cases, we do not know if this is the case. Researchers investigating microsaccades probably have a different (implicit) definition of a saccade than do eye movement researchers investigating gaze behavior to faces.…”

“…Andersson, Larsson, Holmqvist, Stridh, and Nyström (2017) wrote: BA Human-Algorithm comparison, however, often assumes that humans behave perfectly rationally and that, consequently, any deviation from perfect agreement is due to the mistakes of the algorithm^(p. 619). The procedure is often as follows; two or three human coders and the algorithm classify a significant quantity of data.…”

“…Future research should examine whether expert coders serve as a good gold standard for event-detection algorithms^(p. xxx). Even Andersson et al (2017), who applied the gold standard approach, wrote: BHuman coders are not perfect and there are indeed difficult classification cases, but the general sentiment is that, in the simple case, what is a fixation and what is a saccade is something we can agree on^(p. 634).…”

“…We computed Cohen's kappa (Cohen, 1960) for all human coder pairs because many recent comparison studies use this measure (Andersson et al, 2017;Larsson, Nyström, Andersson, & Stridh, 2015;Larsson, Nyström, & Stridh, 2013;Zemblys et al, 2017). Cohen's kappa is a sample-based measure to quantify the agreement between two coders.…”