Eye-tracking glasses in face-to-face interactions: Manual versus automated assessment of areas-of-interest

Abstract

The assessment of gaze behaviour is essential for understanding the psychology of communication. Mobile eye-tracking glasses are useful to measure gaze behaviour during dynamic interactions. Eye-tracking data can be analysed by using manually annotated areas-of-interest. Computer vision algorithms may alternatively be used to reduce the amount of manual effort, but also the subjectivity and complexity of these analyses. Using additional re-identification (Re-ID) algorithms, different participants in the interaction can be distinguished. The aim of this study was to compare the results of manual annotation of mobile eye-tracking data with the results of a computer vision algorithm. We selected the first minute of seven randomly selected eye-tracking videos of consultations between physicians and patients in a Dutch Internal Medicine out-patient clinic. Three human annotators and a computer vision algorithm annotated mobile eye-tracking data, after which interrater reliability was assessed between the areas-of-interest annotated by the annotators and the computer vision algorithm. Additionally, we explored interrater reliability when using lengthy videos and different area-of-interest shapes. In total, we analysed more than 65 min of eye-tracking videos manually and with the algorithm. Overall, the absolute normalized difference between the manual and the algorithm annotations of face-gaze was less than 2%. Our results show high interrater agreements between human annotators and the algorithm with Cohen’s kappa ranging from 0.85 to 0.98. We conclude that computer vision algorithms produce comparable results to those of human annotators. Analyses by the algorithm are not subject to annotator fatigue or subjectivity and can therefore advance eye-tracking analyses.

Immersive Experience and Climate Change Monitoring in Digital Landscapes: Evidence from Somatosensory Sense and Comfort

“…Based on this, this study further investigates subjects’ interest experience in digital landscape roaming scenarios. The change trend in the interest degree of the five groups of virtual landscape scene nodes confirms that the subjects had the highest interest degree in the ancient tree ecological area during the immersion experience, which is consistent with the research of Hessels et al (2016) and Jongerius et al (2021) [ 61 , 62 ]. Wang et al analyzed the correlation between subjects’ interest in AOI.…”

Looking at faces in the wild

“…Our dynamic region of interest (dROI) analysis of social attention relied on automatic face and body detection algorithms developed by Cao and colleagues (OpenPose: 30 ). We verified the accuracy of this algorithm on our video data by comparing its detections to manual coding of body presence by four human observers and found a high level of agreement (see Supplementary Materials Section 1.1 ; see also 34 ).

Figure 1 Dynamic region of interest (dROI) analysis of social attention while navigating a university campus.

…”

Evaluating the integration of eye-tracking and motion capture technologies: Quantifying the accuracy and precision of gaze measures

“…However, these algorithm based approaches are limited when evaluating gaze behaviors in locomotor settings as classifying gaze behaviors based exclusively on eye orientation may not account for disparity between the participants eye and bodily motion ( Hessels et al, 2018 ; Lappi, 2015 ). As such, as a person moves through the environment, determining point of gaze relative to environmental features has predominantly been achieved by frame-by-frame analysis of scene camera footage ( Hessels, Benjamins, et al, 2020 ; Jongerius et al, 2021 ; Kothari et al, 2020 ). In particular, a frame-by-frame approach has been commonly applied to consider the amount of time a person's point of gaze is located on designated environmental locations ( Ellmers et al., 2016 , 2020 ; Mele & Federici, 2012 ; Parr et al, 2020 ).…”

“…However, it has been suggested that frame-by-frame analysis methods are prone to inter-observer error and have been considered time consuming ( Duchowski, 2007 ; Hessels, Niehorster, et al, 2020 ; Kiefer et al, 2017 ). Further, researchers evaluating eye-tracking data analysis have recently noted that eye-tracking research would benefit from more standardized analysis procedures to enhance comparison between studies and assessment of study quality ( Jongerius et al, 2021 ). The capacity of optoelectronic data collection to express both point of gaze and the location of environmental features within a world-centered reference frame addresses these points by facilitating greater automation of the data analysis process, which promotes time efficiency, reduces inter-observer error, and greatly increases the number of trials that can be analyzed.…”

“…The capacity of optoelectronic data collection to express both point of gaze and the location of environmental features within a world-centered reference frame addresses these points by facilitating greater automation of the data analysis process, which promotes time efficiency, reduces inter-observer error, and greatly increases the number of trials that can be analyzed. In turn, these attributes increase statistical power, which due to the time needed to manually code gaze data has been frequently recognized as low ( Jongerius et al, 2021 ; Knudson, 2017 ). In addition to the advantages associated with automated data analysis, the ability to express both point of gaze and environmental features within the laboratory coordinate system allows researchers to evaluate the accuracy and precision of gaze data in terms of world-space and contributes to effective experimental design.…”