Human Sensitivity to Perturbations Constrained by a Model of the Natural Image Manifold

Abstract

Humans are remarkably well tuned to the statistical properties of natural images. However, quantitative characterization of processing within the domain of natural images has been difficult because most parametric manipulations of a natural image make that image appear less natural. We used generative adversarial networks (GANs) to constrain parametric manipulations to remain within an approximation of the manifold of natural images. In the first experiment, 7 observers decided which one of two synthetic perturbed images matched a synthetic unperturbed comparison image. Observers were significantly more sensitive to perturbations that were constrained to an approximate manifold of natural images than they were to perturbations applied directly in pixel space. Trial by trial errors were consistent with the idea that these perturbations disrupt configural aspects of visual structure used in image segmentation. In a second experiment, 5 observers discriminated paths along the image manifold as recovered by the GAN. Observers were remarkably good at this task, confirming that observers were tuned to fairly detailed properties of an approximate manifold of natural images. We conclude that human tuning to natural images is more general than detecting deviations from natural appearance, and that humans have, to some extent, access to detailed interrelations between natural images.

Contrast invariant tuning in human perception of image content

“…Many characterizations of visual performance have been obtained with highly simplistic stimuli, such as gratings [3,4,28,36,27] or simple line drawings [43,44,45]. Parametric measurements of behavioural performance in the high-dimensional and non-linear feature space that defines natural-appearing images is difficult [20]. For example, Sebastian et al [46] superimposed target gratings on natural images resulting in stimuli with artificial foreground and natural background-still a very unnatural combination.…”

“…To address this question, we re-analyzed data from an independent, previously published study [20]. In that study, 8 observers decided which one of two flanking stimuli was closer to a centrally presented comparison.…”

“…Here E[·] denotes expectation, z is a 128-dimensional isotropic standard normal variable, x are images from the training dataset,x is an example image at a random location between the simulated images G(z) and the real images x, and ∇ denotes the gradient operator (with respect tox). The details of the training procedure have been described elsewhere [20].…”

“…Instead of directly comparing processing in a deep artificial neural network to processing in the human visual system, we use a deep artificial neural network to generate naturalistic-looking images. Specifically, we use a generative adversarial network (GAN, [19]) to approximately constrain high-dimensional stimulus manipulations to the domain of natural images [20].…”

“…GANs have recently been successful at generating extremely realistic looking images (see for example, [21,22]). Furthermore, GANs seem to recover a parametrization of the manifold of natural images that matches aspects of human perception [20] and predicts neural responses in non-human primates [23]. By expressing manipulations of naturalistic images along this approximation of the manifold of natural images, we can characterize complex invariances in human visual perception without a need to rely on analogies between human processing and processing in deep artificial neural network algorithms.…”