A parametric texture model based on deep convolutional features closely matches texture appearance for humans

Abstract

Our visual environment is full of texture-"stuff" like cloth, bark, or gravel as distinct from "things" like dresses, trees, or paths-and humans are adept at perceiving subtle variations in material properties. To investigate image features important for texture perception, we psychophysically compare a recent parametric model of texture appearance (convolutional neural network [CNN] model) that uses the features encoded by a deep CNN (VGG-19) with two other models: the venerable Portilla and Simoncelli model and an extension of the CNN model in which the power spectrum is additionally matched. Observers discriminated model-generated textures from original natural textures in a spatial three-alternative oddity paradigm under two viewing conditions: when test patches were briefly presented to the near-periphery ("parafoveal") and when observers were able to make eye movements to all three patches ("inspection"). Under parafoveal viewing, observers were unable to discriminate 10 of 12 original images from CNN model images, and remarkably, the simpler Portilla and Simoncelli model performed slightly better than the CNN model (11 textures). Under foveal inspection, matching CNN features captured appearance substantially better than the Portilla and Simoncelli model (nine compared to four textures), and including the power spectrum improved appearance matching for two of the three remaining textures. None of the models we test here could produce indiscriminable images for one of the 12 textures under the inspection condition. While deep CNN (VGG-19) features can often be used to synthesize textures that humans cannot discriminate from natural textures, there is currently no uniformly best model for all textures and viewing conditions.

Canal–otolith interactions alter the perception of self-motion direction

“…The LOO is a measure of a model's predictive accuracy. In the following comparisons, we report the model-specific LOO information criterion (LOOIC) and the LOOIC difference between the two models, as well as their standard errors of the LOOIC (see also Wallis et al, 2017, for a similar procedure). The LOOIC estimates the expected log pointwise predictive density.…”

Deep Synthesis of Realistic Medical Images: A Novel Tool in Clinical Research and Training

“…During the ensuing DS phase, the fully trained DNN is made to synthesize one or more new images that capture the statistics of the training images. The specific method we used ( Gatys et al, 2015 , 2017 ; Wallis et al, 2017 ) further receives one additional “sample” image and synthesizes counterpart images, i.e., images that capture the statistics of the sample image specifically, rather than the general statistics of all images in the training set.…”

“…The DS method we used is based on previous work by others ( Gatys et al, 2015 , 2017 ; Wallis et al, 2017 ). In this method, the fully trained DNN was presented with a series of additional, actual PVMs (“original mammograms”).…”

Chemometrics and hyperspectral imaging applied to assessment of chemical, textural and structural characteristics of meat

“…As results, it represents all levels of complexity of the hyperspectral data within a single model. For example, Gatys et al introduced the concept of a nonlinear filter bank using convolutional neural networks to describe texture in images (Gatys et al, 2015;Wallis et al, 2017).…”