Measures and Limits of Models of Fixation Selection

Wilming, Niklas; Betz, Torsten; Kietzmann, Tim C.; König, Peter

doi:10.1371/journal.pone.0024038

Cited by 61 publications

(105 citation statements)

References 34 publications

Supporting

Mentioning

103

Contrasting

Order By: Relevance

“…This model is the upper bound on prediction in the dataset (see ref. 8 for a thorough comparison of this gold standard and other upper bounds capturing different constraints).…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Information-theoretic model comparison unifies saliency metrics

Kümmerer

Wallis

Bethge

2015

Proc. Natl. Acad. Sci. U.S.A.

147

167

View full text Add to dashboard Cite

Learning the properties of an image associated with human gaze placement is important both for understanding how biological systems explore the environment and for computer vision applications. There is a large literature on quantitative eye movement models that seeks to predict fixations from images (sometimes termed "saliency" prediction). A major problem known to the field is that existing model comparison metrics give inconsistent results, causing confusion. We argue that the primary reason for these inconsistencies is because different metrics and models use different definitions of what a "saliency map" entails. For example, some metrics expect a model to account for image-independent central fixation bias whereas others will penalize a model that does. Here we bring saliency evaluation into the domain of information by framing fixation prediction models probabilistically and calculating information gain. We jointly optimize the scale, the center bias, and spatial blurring of all models within this framework. Evaluating existing metrics on these rephrased models produces almost perfect agreement in model rankings across the metrics. Model performance is separated from center bias and spatial blurring, avoiding the confounding of these factors in model comparison. We additionally provide a method to show where and how models fail to capture information in the fixations on the pixel level. These methods are readily extended to spatiotemporal models of fixation scanpaths, and we provide a software package to facilitate their use.visual attention | eye movements | probabilistic modeling | likelihood | point processes H umans move their eyes about three times/s when exploring the environment, fixating areas of interest with the highresolution fovea. How do we determine where to fixate to learn about the scene in front of us? This question has been studied extensively from the perspective of "bottom-up" attentional guidance (1), often in a "free-viewing" task in which a human observer explores a static image for some seconds while his or her eye positions are recorded (Fig. 1A). Eye movement prediction is also applied in domains from advertising to efficient object recognition. In computer vision the problem of predicting fixations from images is often referred to as "saliency prediction," while to others "saliency" refers explicitly to some set of low-level image features (such as edges or contrast). In this paper we are concerned with predicting fixations from images, taking no position on whether the features that guide eye movements are "low" or "high" level.The field of eye movement prediction is quite mature: Beginning with the influential model of Itti et al. (1), there are now over 50 quantitative fixation prediction models, including around 10 models that seek to incorporate "top-down" effects (see refs. 2-4 for recent reviews and analyses of this extensive literature). Many of these models are designed to be biologically plausible whereas others aim purely at prediction (e.g., ref. 5). Progress is meas...

show abstract

“…This model is the upper bound on prediction in the dataset (see ref. 8 for a thorough comparison of this gold standard and other upper bounds capturing different constraints).…”

Section: Methodsmentioning

confidence: 99%

“…To answer this question requires a principled distance metric, yet no such metric exists. There is significant uncertainty about how to compare saliency models (3,(6)(7)(8). A visit to the well-established MIT Saliency Benchmark (saliency.mit.edu) allows the reader to order models by seven different metrics.…”

mentioning

confidence: 99%

Information-theoretic model comparison unifies saliency metrics

Kümmerer

Wallis

Bethge

2015

Proc. Natl. Acad. Sci. U.S.A.

147

167

View full text Add to dashboard Cite

show abstract

“…The present work attempted to deal with these issues by averaging fixation positions over trials and observers [47] and comparing the regressions of fixation position and central viewing bias [12,31,64] on local scene color defined by all three variables, lightness and the red-green and yellow-blue chromatic components, with appropriate allowance for the number of explanatory variables in the fits.…”

Section: Discussionmentioning

confidence: 99%

“…Fixation data from each scene were pooled over observers to reflect the systematic effects of scene structure and viewing bias [47] rather than random variations between observers [21].…”

Section: E Gaze Analysismentioning

confidence: 99%

“…These included the use of the information-theoretic divergence of distributional differences, which loses the connection to a simple regression model, and the use of the area under a receiver operating characteristic (ROC) curve as the performance measure, which requires a classificatory approach [16,47]. These measures were not taken further, although values of the area under the ROC curve (AUC) were estimated for the four example scenes to allow comparison with the R 2 estimates.…”

Section: Accounting For Variancementioning

confidence: 99%

See 1 more Smart Citation

Influence of local scene color on fixation position in visual search

Amano

Foster

2014

J. Opt. Soc. Am. A

View full text Add to dashboard Cite

Where observers concentrate their gaze during visual search depends on several factors. The aim here was to determine how much of the variance in observers' fixations in natural scenes can be explained by local scene color and how that variance is related to viewing bias. Fixation data were taken from an experiment in which observers searched images of 20 natural rural and urban scenes for a small target. The proportion R 2 of the variance explained in a regression on local color properties (lightness and the red-green and yellow-blue chromatic components) ranged from 1% to 85%, depending mainly on how well those properties were consistent with observers' viewing bias. When viewing bias was included in the regression, values of R 2 increased, ranging from 62% to 96%. By comparison, local lightness and local lightness contrast, edge density, and entropy each explained less variance than local color properties. Local scene color may have a much stronger influence on gaze position than is generally recognized, capturing significant aspects of scene structure on target search behavior.

show abstract

Saliency Benchmarking Made Easy: Separating Models, Maps and Metrics

Kümmerer

Wallis

Bethge

2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Dozens of new models on fixation prediction are published every year and compared on open benchmarks such as MIT300 and LSUN. However, progress in the field can be difficult to judge because models are compared using a variety of inconsistent metrics. Here we show that no single saliency map can perform well under all metrics. Instead, we propose a principled approach to solve the benchmarking problem by separating the notions of saliency models, maps and metrics. Inspired by Bayesian decision theory, we define a saliency model to be a probabilistic model of fixation density prediction and a saliency map to be a metric-specific prediction derived from the model density which maximizes the expected performance on that metric given the model density. We derive these optimal saliency maps for the most commonly used saliency metrics (AUC, sAUC, NSS, CC, SIM, KL-Div) and show that they can be computed analytically or approximated with high precision.We show that this leads to consistent rankings in all metrics and avoids the penalties of using one saliency map for all metrics. Our method allows researchers to have their model compete on many different metrics with state-of-the-art in those metrics: "good" models will perform well in all metrics.

show abstract

Measures and Limits of Models of Fixation Selection

Cited by 61 publications

References 34 publications

Information-theoretic model comparison unifies saliency metrics

Information-theoretic model comparison unifies saliency metrics

Influence of local scene color on fixation position in visual search

Saliency Benchmarking Made Easy: Separating Models, Maps and Metrics

Contact Info

Product

Resources

About