Human sensitivity to perturbations constrained by a model of the natural image manifold

Abstract

Humans are remarkably well tuned to the statistical properties of natural images. However, quantitative characterization of processing within the domain of natural images has been difficult because most parametric manipulations of a natural image make that image appear less natural. We used generative adversarial networks (GANs) to constrain parametric manipulations to remain within an approximation of the manifold of natural images. In the first experiment, seven observers decided which one of two synthetic perturbed images matched a synthetic unperturbed comparison image. Observers were significantly more sensitive to perturbations that were constrained to an approximate manifold of natural images than they were to perturbations applied directly in pixel space. Trial-bytrial errors were consistent with the idea that these perturbations disrupt configural aspects of visual structure used in image segmentation. In a second experiment, five observers discriminated paths along the image manifold as recovered by the GAN. Observers were remarkably good at this task, confirming that observers are tuned to fairly detailed properties of an approximate manifold of natural images. We conclude that human tuning to natural images is more general than detecting deviations from natural appearance, and that humans have, to some extent, access to detailed interrelations between natural images.

Using generative models of naturalistic scenes to sample neural population tuning manifolds

“…We hoped to bridge this gap with the use of a GAN trained on natural images. These GANs provide rich, colorful, and complex stimuli (as shown in Figure 1) while maintaining natural world image statistics (Fruend and Stalker, 2018), and providing a lower-dimensional parameter space (in our case, 128 latent dimensions).…”

“…GANs trained to generate images learned to map high-order image statistics onto a set of lowerdimensional latent variables. The GAN used in our current experiments had a 128dimensional input (latent) space, and was trained on the CIFAR-10 image data set (Krizhevsky, 2009) as described previously (Fruend and Stalker, 2018). Example GANgenerated images can be found in Figure 1.…”

“…Thus, if a human observer judges one compression method to discard more psychologically relevant information than the other, that likely reflects that one of the compression methods more closely resembles the human perceptual system. Indeed, a rigorous psychophysical study with the identical image model showed human observers are most sensitive to manipulations of images in the GAN latent space, compared to similarmagnitude manipulations in pixel-space or Fourier-space, suggesting that the features captured by the GAN reflect fundamental neural computations (Fruend and Stalker, 2018).…”

Application of ELM algorithm-based generative adversarial network model in network security

“…Generative adversarial networks, abbreviated as GAN, are a framework for training and evaluating generative models [9][10][11]. In this framework, two models are trained simultaneously: a generative model G and a discriminative model D [12][13][14].…”

Constrained sampling from deep generative image models reveals mechanisms of human target detection

“…One challenge here is the difficulty of generating and manipulating stimuli with naturalistic variability in appearance. Modern generative image models based on artificial neural networks, such as Generative Adversarial Networks ( GANs; Goodfellow et al., 2014 ), can generate impressively realistic images (see examples in Brock et al., 2018 , Miyato et al., 2018 ), and human visual performance is highly sensitive to image manipulations guided by these models ( Fruend & Stalker, 2018 ).…”