Using machine learning to detect events in eye-tracking data

Zemblys, Raimondas; Niehorster, Diederick C; Komogortsev, Oleg V.; Holmqvist, Kenneth

doi:10.3758/s13428-017-0860-3

Cited by 141 publications

(130 citation statements)

References 32 publications

Supporting

Mentioning

105

Contrasting

Unclassified

Order By: Relevance

“…Although Komogortsev, Gobert, Jayarathna, Koh, and Gowda (2010) have previously written about manual classification that In fields were human classification until recently dominated, automated algorithms take over quickly. New classification techniques such as identification by topological characteristics (Hein & Zangemeister, 2017) and machine learning (Zemblys et al, 2017) are promising. Other new algorithms (based on classic techniques) can deal with smooth pursuit episodes (Larsson et al, 2015) or a large variety of noise levels (Hessels, Niehorster, et al, 2016b).…”

Section: Labels and The Role Of Instructions And Event Selection Rulesmentioning

confidence: 99%

“…Manual classification to teach artificial intelligence (AI) and to develop algorithms Zemblys et al (2017) wrote: BAny already manually or algorithmically detected events can be used to train a classifier to produce similar classification of other data without the need for a user to set parameters^. It would be interesting if machine learning is used to produce automated AI classifiers that have the ability to classify eye-tracking data for which no classical algorithm exists.…”

Section: Labels and The Role Of Instructions And Event Selection Rulesmentioning

confidence: 99%

“…We computed Cohen's kappa (Cohen, 1960) for all human coder pairs because many recent comparison studies use this measure (Andersson et al, 2017;Larsson, Nyström, Andersson, & Stridh, 2015;Larsson, Nyström, & Stridh, 2013;Zemblys et al, 2017). Cohen's kappa is a sample-based measure to quantify the agreement between two coders.…”

Section: Agreement Measuresmentioning

confidence: 99%

See 2 more Smart Citations

Is human classification by experienced untrained observers a gold standard in fixation detection?

et al. 2017

Self Cite

View full text Add to dashboard Cite

Manual classification is still a common method to evaluate event detection algorithms. The procedure is often as follows: Two or three human coders and the algorithm classify a significant quantity of data. In the gold standard approach, deviations from the human classifications are considered to be due to mistakes of the algorithm. However, little is known about human classification in eye tracking. To what extent do the classifications from a larger group of human coders agree? Twelve experienced but untrained human coders classified fixations in 6 min of adult and infant eye-tracking data. When using the sample-based Cohen's kappa, the classifications of the humans agreed near perfectly. However, we found substantial differences between the classifications when we examined fixation duration and number of fixations. We hypothesized that the human coders applied different (implicit) thresholds and selection rules. Indeed, when spatially close fixations were merged, most of the classification differences disappeared. On the basis of the nature of these intercoder differences, we concluded that fixation classification by experienced untrained human coders is not a gold standard. To bridge the gap between agreement measures (e.g., Cohen's kappa) and eye movement parameters (fixation duration, number of fixations), we suggest the use of the event-based F1 score and two new measures: the relative timing offset (RTO) and the relative timing deviation (RTD).

show abstract

Section: Labels and The Role Of Instructions And Event Selection Rulesmentioning

confidence: 99%

Section: Labels and The Role Of Instructions And Event Selection Rulesmentioning

confidence: 99%

See 1 more Smart Citation

Is human classification by experienced untrained observers a gold standard in fixation detection?

et al. 2017

Self Cite

View full text Add to dashboard Cite

show abstract

“…Of the few algorithm that are capable of detecting saccades as well as PSO 31,32,42 , U'n'Eye achieves highest performance. Note that Zemblys et al 42 reported higher absolute Cohen's kappa values for saccade and PSO detection, but obtained these on cleaner data than the benchmark dataset from Andersson et al 29 . As their dataset was not available to us, we were not able to assess U'n'Eye's performance on the same data.…”

Section: Discussionmentioning

confidence: 99%

Human-level saccade detection performance using deep neural networks

Bellet

Nienborg

et al. 2018

Preprint

View full text Add to dashboard Cite

Saccades are ballistic eye movements that rapidly shift gaze from one location of visual space to another. Detecting saccades in eye movement recordings is important not only for studying the neural mechanisms underlying sensory, motor, and cognitive processes, but also as a clinical and diagnostic tool. However, automatically detecting saccades can be difficult, particularly when such saccades are generated in coordination with other tracking eye movements, like smooth pursuits, or when the saccade amplitude is close to eye tracker noise levels, like with microsaccades. In such cases, labeling by human experts is required, but this is a tedious task prone to variability and error. We developed a convolutional neural network (CNN) to automatically detect saccades at human-level performance accuracy. Our algorithm surpasses state of the art according to common performance metrics, and will facilitate studies of neurophysiological processes underlying saccade generation and visual processing.

show abstract

“…To remove erroneous labels, we adapt the approach proposed by Zemblys et al 20 For our dataset, fixational events with <0.5 • separation between them and within 75ms of each other were combined into a single event. Fixations less than 50ms and saccades greater than 150ms in duration were automatically removed.…”

Section: Data Cleaning and Post-processingmentioning

confidence: 99%

Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities

Kothari

Yang²,

Kanan³

et al. 2020

Sci Rep

View full text Add to dashboard Cite

The interaction between the vestibular and ocular system has primarily been studied in controlled environments. Consequently, off-the shelf tools for categorization of gaze events (e.g. fixations, pursuits, saccade) fail when head movements are allowed. Our approach was to collect a novel, naturalistic, and multimodal dataset of eye+head movements when subjects performed everyday tasks while wearing a mobile eye tracker equipped with an inertial measurement unit and a 3D stereo camera. This Gaze-in-the-Wild dataset (GW) includes eye+head rotational velocities (deg/s), infrared eye images and scene imagery (RGB+D). A portion was labelled by coders into gaze motion events with a mutual agreement of 0.72 sample based Cohen's κ. This labelled data was used to train and evaluate two machine learning algorithms, Random Forest and a Recurrent Neural Network model, for gaze event classification. Assessment involved the application of established and novel event based performance metrics. Classifiers achieve ∼90% human performance in detecting fixations and saccades but fall short (60%) on detecting pursuit movements. Moreover, pursuit classification is far worse in the absence of head movement information. A subsequent analysis of feature significance in our best-performing model revealed a reliance upon absolute eye and head velocity, indicating that classification does not require spatial alignment of the head and eye tracking coordinate systems. The GW dataset, trained classifiers and evaluation metrics will be made publicly available with the intention of facilitating growth in the emerging area of head-free gaze event classification. arXiv:1905.13146v1 [cs.CV] 9 May 2019 (see Section 3). We then use this labelled data for supervised training and assessment of automated event detectors.This work builds upon a variety of techniques previously used to track head orientation during natural behavior. Published studies have demonstrated the use of rotational potentiometers and accelerometers, 8 magnetic coils, 12 or motion capture 13 for the sensing of head orientation. 6 Perhaps the highest precision eye+head tracker which allowed body movement leveraged a 5.8 m 3 custom-made armature capable of generating a pulsing magnetic field. The subject was outfitted with a head-worn receiver capable of measuring head position and orientation within its operational region. 14 Several systems have adopted video based head motion compensation 10, 15 and demonstrated promising results, but are too computationally expensive for real-time use, and are prone to irrecoverable track loss especially during periods of rapid head movement, occlusion of tracking features or degration of image quality due to motion blur. Recent approaches have involved the use of head-mounted IMUs. For example, Larsson et al. used a head-mounted IMU in a study where subjects were asked to perform visual tracking tasks when watching pre-rendered stimuli projected onto a 2D screen. 16 They established that compensating for head movements results in a reduced sta...

show abstract

Using machine learning to detect events in eye-tracking data

Cited by 141 publications

References 32 publications

Is human classification by experienced untrained observers a gold standard in fixation detection?

Is human classification by experienced untrained observers a gold standard in fixation detection?

Human-level saccade detection performance using deep neural networks

Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities

Contact Info

Product

Resources

About