A parametric texture model based on deep convolutional features closely matches texture appearance for humans

Abstract

Our visual environment is full of texture-"stuff" like cloth, bark or gravel as distinct from "things" like dresses, trees or paths-and humans are adept at perceiving subtle variations in material properties. To investigate image features important for texture perception, we psychophysically compare a recent parameteric model of texture appearance (CNN model) that uses the features encoded by a deep convolutional neural network with two other models: the venerable Portilla and Simoncelli model (PS) and an extension of the CNN model in which the power spectrum is additionally matched. Observers discriminated model-generated textures from original natural textures in a spatial three-alternative oddity paradigm under two viewing conditions: when test patches were briefly presented to the near-periphery ("parafoveal") and when observers were able to make eye movements to all three patches ("inspection"). Under parafoveal viewing, observers were unable to discriminate 10 of 12 original images from CNN model images, and remarkably, the simpler PS model performed slightly better than the CNN model (11 textures). Under foveal inspection, matching CNN features captured appearance substantially better than the PS model (9 compared to 4 textures), and including the power spectrum improved appearance matching for two of the three remaining textures. None of the models we test here could produce indiscriminable images for one of the 12 textures under the inspection condition. While deep CNN (VGG-19) features can often be used to synthesise textures that humans cannot discriminate from natural textures, there is currently no uniformly best model for all textures and viewing conditions. Keywords: spatial vision, natural scenes, texture perception, peripheral vision . CC-BY 4.0 International license not peer-reviewed) is the author/funder. It is made available under a The copyright holder for this preprint (which was . http://dx.doi.org/10.1101/165761 doi: bioRxiv preprint first posted online IntroductionTextures are characterised by the repetition of smaller elements, sometimes with variation, to make up a pattern. Significant portions of the visual environment can be thought of as textures ("stuff" as distinct from "things"; Adelson & Bergen, 1991): your neighbour's pink floral wallpaper, the internal structure of dark German bread, the weave of a wicker basket, the gnarled bark of an old tree trunk, a bowl full of prawns ready for the barbie. Texture is an important material property whose perception is of adaptive value (Adelson, 2001;Fleming, 2014). For example, we can readily discriminate wet from dry stones (e.g. Ho, Landy, & Maloney, 2008), separating the underlying spatial texture from potentially temporary characteristics like glossiness. Where surfaces of different textures form occlusion boundaries, texture can provide a powerful segmentation cue; conversely, occlusion borders of similarly-textured surfaces can camoflage the occlusion (hiding a tiger among the leaves). Given the importance and ubiquity of visu...

Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks

“…The current work was motivated by the broad scientific goal of discovering models that quantitatively explain the neuronal mechanisms underlying primate invariant object recognition behavior. To this end, previous work had shown that specific artificial neural network models (ANNs), drawn from a large family of deep convolutional neural networks (DCNNs) and optimized to achieve high levels of object categorization performance on large-scale image-sets, capture a large fraction of the variance in primate visual recognition behaviors (Rajalingham et al, 2015; Jozwik et al, 2016; Kheradpisheh et al, 2016; Kubilius et al, 2016; Peterson et al, 2016; Wallis et al, 2017), and the internal hidden neurons of those same models also predict a large fraction of the image-driven response variance of brain activity at multiple stages of the primate ventral visual stream (Yamins et al, 2013; Cadieu et al, 2014; Khaligh-Razavi and Kriegeskorte, 2014; Yamins et al, 2014; Güçlü and van Gerven, 2015; Cichy et al, 2016; Hong et al, 2016; Seibert et al, 2016; Cadena et al, 2017; Wen et al, 2017). For clarity, we here referred to this sub-family of models as DCNN IC (to denote ImageNet-Categorization training), so as to distinguish them from all possible models in the DCNN family, and more broadly, from the super-family of all ANNs.…”

Human Sensitivity to Perturbations Constrained by a Model of the Natural Image Manifold

“…We therefore believe that our approach is complementary to the local approach taken by studies that investigated texture processing (e.g. Freeman & Simoncelli, 2011;Gerhard et al, 2013;Wallis et al, 2016Wallis et al, , 2017 Pixels Fourier Latent Image domain 1 0 1 2 Discriminability d Figure 7: Turns in other image spaces are harder to detect Average sensitivity for detecting a 90-degree turn in an image sequence along a path through pixel space, through Fourier space, and through latent space. Error bars mark 95% confidence intervals and gray dots mark performance of individual observers.…”

“…Our images were only 32×32 pixels in size. In contrast, Wallis et al (2017) used image patches that were 128×128 pixels to compare between texture images created from a deep neural network model and real photographs of textures. Other studies have used a range of images sizes (Alam et al, 2014;Sebastian et al, 2017;Bex, 2010, in increasing order of image size), but our images are closer to the range of image sizes used as patches of images (Gerhard et al, 2013) rather than entire images.…”

“…Although each one of the examples can clearly be segmented into foreground and background, it is not always possible to actually name the foreground objects in Figure 1B., while this seems to be easier for the training examples in Figure 1A., despite the relatively low resolution of the images. Thus, the samples from the GAN used in the present study would probably be easy to discriminate from real images if they were directly compared to real images (Gerhard et al, 2013;Wallis et al, 2017). It should thus be noted that the image representation learned by the GAN used in this study is only an approximation to the manifold of natural images (note however, that other studies training on the CIFAR10 dataset show samples of similar quality, for example Gulrajani et al, 2017;Roth et al, 2017).…”

An Artificial Intelligence-based Multimodality-Integration for Precise Prediction of Prostate Cancer Lymph Node Metastasis: A Retrospective Two-center Study

“…We further investigated a tumor-related region around the PCa to extract deep learning-embedded features. The deep learningembedded features adopted a transfer learning technique based on a VGG-19 networks which is pretrained on ImageNet 28 . For each case, the axis-slice with largest area of the PCa cross section was selected from axial T 2 WI, high-b DWI and ADC images.…”