The Mental Image Revealed by Gaze Tracking

Wang, Xi; Ley, Andreas; Koch, Sebastian; Lindlbauer, David; Hays, James; Holmqvist, Kenneth; Alexa, Marc

doi:10.1145/3290605.3300839

Cited by 14 publications

(9 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The retrieval accuracy of both CNN and kNN were significantly above chance (AUC = 50% ). Similar to previous results 25 , CNN achieved the accuracy of AUC = 97.5% and AUC = 94.5% was achieved when using kNN. The achieved top-1 and top-3 ranks using CNN were 61.3% and 79.1% respectively, and 60.6% and 77.9% using kNN.…”

Section: Characteristics Of Eye Movements On Real and Mental Images supporting

confidence: 87%

“…For comparison, we employed a convolutional neural network (CNN) following the structure proposed in Ref. 25 .…”

Section: Characteristics Of Eye Movements On Real and Mental Images mentioning

confidence: 99%

“…To investigate the generalizability of the system, we used two networks similar to the ones used for classification to learn 16-dimensional descriptors from the eye movement histograms. Following the architecture proposed in 25 , we used triplet loss 27 to force descriptors of matching encoding and recall gaze histograms to be closer together than non-matching ones, without learning to map between them. The resulting dual networks allowed us to see whether retrieval based on eye movements during mental imagery can generalize to new image content that is not in the training dataset used by the networks.…”

Section: Generalized Retrieval Of New Image Contentmentioning

confidence: 99%

“…Each encoding (or, respectively, recall) sequence of eye movements was accumulated into a histogram which served as a one channel input image to the CNN. The classification network was similar to the architecture used in 25 . In the dual-task network (Fig.…”

Section: Cnn Classification For Image Retrieval From Encoding or Recmentioning

confidence: 99%

See 3 more Smart Citations

Computational discrimination between natural images based on gaze during mental imagery

Wang

Ley

Koch

et al. 2020

Sci Rep

Self Cite

View full text Add to dashboard Cite

When retrieving image from memory, humans usually move their eyes spontaneously as if the image were in front of them. Such eye movements correlate strongly with the spatial layout of the recalled image content and function as memory cues facilitating the retrieval procedure. However, how close the correlation is between imagery eye movements and the eye movements while looking at the original image is unclear so far. In this work we first quantify the similarity of eye movements between recalling an image and encoding the same image, followed by the investigation on whether comparing such pairs of eye movements can be used for computational image retrieval. our results show that computational image retrieval based on eye movements during spontaneous imagery is feasible. furthermore, we show that such a retrieval approach can be generalized to unseen images. Research on eye movements during visual imagery has a long history. Early investigations 1-3 from the beginning of the twentieth century have established a link between eye movements during visual imagery and mental images, based on observations of eye movements during imagery. Neisser 4 argued that eye movements are actively associated with the construction of a visual image, and Hebb 5 suggested that eye movements during imagery and memory retrieval are necessary to assemble and organize "part images" into a whole visualized image. Many recent studies 6-14 have shown that humans spontaneously move their eyes when recalling a scene from memory and that such eye movement patterns closely resemble the spatial arrangement of the elements of the recalled content. This effect has not only been demonstrated for participants who encode visual scenes and later recall those scenes while looking at an empty screen 6,7,9,11,14,15 , but also for participants who encode verbal information in association with a spatial cue and later recall such information while looking at an empty screen 12,13. Moreover, it has been shown that participants who listen to scene descriptions also make eye movements which correspond to spatial positions from the described scene 7,14,15. Previous studies evaluated the functionality of eye movements to blank space. Results suggest that eye movements while looking at nothing play a functional role during memory retrieval, and act as "spatial indices" 4,5,11,12,16 that may provide assistance in memorizing the spacial layout of a scene 4-6,11. However, it has been argued whether such eye movements can facilitate memory retrieval as additional cues 17,18. In support of such a functional role, impaired episodic memory performance has been reported in experiments where participants were restricted to look at a fixation cross during recall 9-11,15. In Johansson et al. 9 , participants' gaze directions were manipulated towards positions on an empty screen that are either overlapped with the original locations of the to-be-retrieved visuospatial information or selected randomly at irrelevant locations. Results demonstrated that the likelihood of succes...

show abstract

Section: Characteristics Of Eye Movements On Real and Mental Images supporting

confidence: 87%

“…For comparison, we employed a convolutional neural network (CNN) following the structure proposed in Ref. 25 .…”

Section: Characteristics Of Eye Movements On Real and Mental Images mentioning

confidence: 99%

Section: Generalized Retrieval Of New Image Contentmentioning

confidence: 99%

Section: Cnn Classification For Image Retrieval From Encoding or Recmentioning

confidence: 99%

See 2 more Smart Citations

Computational discrimination between natural images based on gaze during mental imagery

Wang

Ley

Koch

et al. 2020

Sci Rep

Self Cite

View full text Add to dashboard Cite

show abstract

“…An example of insufficient data quality would be if the eye-tracking setup records gaze positions with a systematic offset that is too large to reliably distinguish which of two nearby objects of interest an observer looks at (Orquin & Holmqvist 2018;Hessels, Kemner, van den Boomen, & Hooge 2016), such as for instance different facial features. Eye-tracker manufacturers often limit themselves to reporting data-quality measurements for well-controlled and optimal scenarios, and many researchers simply reiterate the manufacturer's statements as applicable to their studies instead of assessing data quality of the eyetracking setup as used in their study (see e.g., Wang et al 2019;Freeth & Bugembe 2018;Hoppe, Loetscher, Morey, & Bülling, 2018). Also a recent overview of head-worn eye-tracking setups (Cognolato, Atzori, & Müller, 2018) compared eye-tracker performance based on manufacturerprovided data-quality values instead of measuring these values themselves.…”

Section: Introductionmentioning

confidence: 99%

The impact of slippage on the data quality of head-worn eye trackers

et al. 2020

View full text Add to dashboard Cite

Mobile head-worn eye trackers allow researchers to record eye-movement data as participants freely move around and interact with their surroundings. However, participant behavior may cause the eye tracker to slip on the participant's head, potentially strongly affecting data quality. To investigate how this eye-tracker slippage affects data quality, we designed experiments in which participants mimic behaviors that can cause a mobile eye tracker to move. Specifically, we investigated data quality when participants speak, make facial expressions, and move the eye tracker. Four head-worn eye-tracking setups were used: (i) Tobii Pro Glasses 2 in 50 Hz mode, (ii) SMI Eye Tracking Glasses 2.0 60 Hz, (iii) Pupil-Labs' Pupil in 3D mode, and (iv) Pupil-Labs' Pupil with the Grip gaze estimation algorithm as implemented in the EyeRecToo software. Our results show that whereas gaze estimates of the Tobii and Grip remained stable when the eye tracker moved, the other systems exhibited significant errors (0.8-3.1 • increase in gaze deviation over baseline) even for the small amounts of glasses movement that occurred during the speech and facial expressions tasks. We conclude that some of the tested eye-tracking setups may not be suitable for investigating gaze behavior when high accuracy is required, such as during face-to-face interaction scenarios. We recommend that users of mobile head-worn eye trackers perform similar tests with their setups to become aware of its characteristics. This will enable researchers to design experiments that are robust to the limitations of their particular eye-tracking setup. Keywords Head-mounted eye tracking • Wearable eye tracking • Mobile eye tracking • Eye movements • Natural behavior • Data quality Diederick C. Niehorster and Thiago Santini contributed equally and should be considered co-first authors.

show abstract