There is no evidence that meaning maps capture semantic information relevant to gaze guidance: Reply to Henderson, Hayes, Peacock, and Rehrig (2021)

Semantic object-scene inconsistencies affect eye movements, but not in the way predicted by contextualized meaning maps

“…A recent study from our group evaluated meaning maps by comparing them to a wider range of saliency models. The results highlight some limitations of the method ( Pedziwiatr et al, 2021a ; see Henderson et al, 2021 and Pedziwiatr, Kümmerer, M., Wallis, Bethge, & Teufel, 2021b for ongoing debate). One key finding of this study demonstrates that meaning maps are outperformed in predicting fixations by DeepGaze II ( Kümmerer et al, 2016 ; Kümmerer et al, 2017 ), a saliency model based on a deep neural network, that indexes high-level features rather than meaning.…”

“…One key finding of this study demonstrates that meaning maps are outperformed in predicting fixations by DeepGaze II ( Kümmerer et al, 2016 ; Kümmerer et al, 2017 ), a saliency model based on a deep neural network, that indexes high-level features rather than meaning. We interpreted this result to suggest that there is so far no evidence that meaning maps measure semantic information that is relevant for gaze guidance (however, see counterpoints in Henderson et al, 2021 , and response in Pedziwiatr et al, 2021b ). Rather, they might index those visual features that are often correlated with semantics, similar to modern saliency models.…”

Knowledge-driven perceptual organization reshapes information sampling via eye movements

“…More generally, a limitation of many previous studies that aim to show the contribution of objects, or semantic meaning to oculomotor control is their reliance on a comparison to computational models that calculate image-computable feature maps as their null hypothesis. The typical approach (Pedziwiatr, Kümmerer, Wallis, Bethge, & Teufel, 2021b) of these studies is (i) to compute a saliency map based on certain features of images used in the experiment, (ii) to generate a map of semantically important regions or object locations in these images, and (iii) to assess which of the two maps better predicts human fixations (for example, see Henderson & Hayes, 2017;Pilarczyk & Kuniecki, 2014;Rider et al, 2018). Insofar as one of these two maps better predicts human fixations, that factor is considered to be critical in gaze guidance.…”

“…A similar dispute regarding the usefulness of meaning maps serves as another example: the seminal meaning maps study claimed that eye-movements are driven by meaning rather than image-computable features because meaning maps were better at predicting fixations than one specific saliency model -the GBVS model (Harel et al, 2007). However, subsequent work showed that meaning maps are outperformed by more advanced saliency models that are based on image-computable features, such as DeepGaze II , challenging the initial interpretation (Henderson, Hayes, Peacock, & Rehrig, 2021;Pedziwiatr et al, 2021aPedziwiatr et al, , 2021b. Independently of the favoured interpretation of these findings, there is a more fundamental aspect that is easily overlooked.…”

“…It is made available under a preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in The copyright holder for this this version posted September 24, 2021. ; https://doi.org/10.1101/2021.09.24.461220 doi: bioRxiv preprint Pedziwiatr, von dem Hagen, & Teufel 5 considered to be critical in gaze guidance. This approach has led to important insights regarding oculomotor control but is hampered by its dependence on saliency models, with the specific choice of model being critical in determining the interpretation (Borji et al, 2013;Pedziwiatr et al, 2021b;Pedziwiatr, Kümmerer, Wallis, Bethge, & Teufel, 2021a). For instance, the conclusion that objects per se, rather than image-computable features that are correlated with objects, guide human eye-movements, as suggested by Einhäuser and colleagues ( 2008), was based on a comparison between manually labelled objects maps and a map derived from one of the earliest saliency models (Itti & Koch, 2000).…”