Visual Analytics and Annotation of Pervasive Eye Tracking Video

Kurzhals, Kuno; Rodrigues, Nils; Koch, Maurice; Stoll, Michael; Bruhn, Andrés; Bulling, Andreas; Weiskopf, Daniel

doi:10.1145/3379155.3391326

Cited by 8 publications

(4 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Silva Machado et al (2019) matched the detected bounding boxes with participants' fixations using a sliding-window approach with a MobileNet CNN model. Rather than using the model for inference on large data for automatic analysis, several tools are multitask and offer interactive visualization for manual annotation (Barz et al, 2023;Kurzhals, 2021;Kurzhals et al, 2017Kurzhals et al, , 2020Panetta et al, 2019). Kurzhals et al (2017) an interactive labeling tool with automatic clustering combined with an analysis system.…”

Section: Deep Learning For Eye Tracking Data Analysismentioning

confidence: 99%

“…Kurzhals et al (2017) an interactive labeling tool with automatic clustering combined with an analysis system. They went on to develop image-based (Kurzhals et al, 2020) and gaze patches techniques (Kurzhals, 2021) for dynamic AOI annotation that are conceptually similar to our proposed idea of merging gaze data with video. Barz et al (2023) have implemented an approach based both on image classification and object detection.…”

Section: Deep Learning For Eye Tracking Data Analysismentioning

confidence: 99%

See 1 more Smart Citation

Quantifying dwell time with location-based augmented reality: Dynamic AOI analysis on mobile eye tracking data with vision transformer

Mercier,

Ertz,

Bocher

2024

JEMR

View full text Add to dashboard Cite

Mobile eye tracking captures egocentric vision and is well-suited for naturalistic studies. However, its data is noisy, especially when acquired outdoor with multiple participants over several sessions. Area of interest analysis on moving targets is difficult because A) camera and objects move nonlinearly and may disappear/reappear from the scene; and B) off-the-shelf analysis tools are limited to linearly moving objects. As a result, researchers resort to time-consuming manual annotation, which limits the use of mobile eye tracking in naturalistic studies. We introduce a method based on a fine-tuned Vision Transformer (ViT) model for classifying frames with overlaying gaze markers. After fine-tuning a model on a manually labelled training set made of 1.98% (=7845 frames) of our entire data for three epochs, our model reached 99.34% accuracy as evaluated on hold-out data. We used the method to quantify participants’ dwell time on a tablet during the outdoor user test of a mobile augmented reality application for biodiversity education. We discuss the benefits and limitations of our approach and its potential to be applied to other contexts.

show abstract

Section: Deep Learning For Eye Tracking Data Analysismentioning

confidence: 99%

Section: Deep Learning For Eye Tracking Data Analysismentioning

confidence: 99%

Quantifying dwell time with location-based augmented reality: Dynamic AOI analysis on mobile eye tracking data with vision transformer

Mercier,

Ertz,

Bocher

2024

JEMR

View full text Add to dashboard Cite

show abstract

“…Multi-layered timelines with spectrograms for different feature intensities were proposed for long-term eye tracking recordings by Kurzhals et al [2020]. By querying selected time spans, similar sequences could be searched and were depicted by short thumbnail sequences of the recorded field of view with a gaze overlay.…”

Section: Mobile Eye Tracking Analysismentioning

confidence: 99%

A Spiral into the Mind: Gaze Spiral Visualization for Mobile Eye Tracking

Koch,

Weiskopf,

Kurzhals

2022

Preprint

Self Cite

View full text Add to dashboard Cite

“…In the context of human-robot interaction, Ramirez-Amaro et al [ 48 ] showed that human behavior inference benefits from incorporating mobile eye tracking data with third person videos. Recently, Kurzhals et al [ 49 ] described an interactive approach for annotating and interpreting egocentric eye tracking data for activity and behavior analysis. They implement an iterative time sequence search based on eye movements and visual features.…”

Section: Related Workmentioning

confidence: 99%

Automatic Visual Attention Detection for Mobile Eye Tracking Using Pre-Trained Computer Vision Models and Human Gaze

Barz

Sonntag

2021

Sensors

View full text Add to dashboard Cite

Processing visual stimuli in a scene is essential for the human brain to make situation-aware decisions. These stimuli, which are prevalent subjects of diagnostic eye tracking studies, are commonly encoded as rectangular areas of interest (AOIs) per frame. Because it is a tedious manual annotation task, the automatic detection and annotation of visual attention to AOIs can accelerate and objectify eye tracking research, in particular for mobile eye tracking with egocentric video feeds. In this work, we implement two methods to automatically detect visual attention to AOIs using pre-trained deep learning models for image classification and object detection. Furthermore, we develop an evaluation framework based on the VISUS dataset and well-known performance metrics from the field of activity recognition. We systematically evaluate our methods within this framework, discuss potentials and limitations, and propose ways to improve the performance of future automatic visual attention detection methods.

show abstract

Visual Analytics and Annotation of Pervasive Eye Tracking Video

Cited by 8 publications

References 37 publications

Quantifying dwell time with location-based augmented reality: Dynamic AOI analysis on mobile eye tracking data with vision transformer

Quantifying dwell time with location-based augmented reality: Dynamic AOI analysis on mobile eye tracking data with vision transformer

A Spiral into the Mind: Gaze Spiral Visualization for Mobile Eye Tracking

Automatic Visual Attention Detection for Mobile Eye Tracking Using Pre-Trained Computer Vision Models and Human Gaze

Contact Info

Product

Resources

About