Representation of Color, Form, and their Conjunction across the Human Ventral Visual Pathway

Abstract

Despite decades of neuroscience research, our understanding of the relationship between color and form processing in the primate ventral visual pathway remains incomplete. Using fMRI multivoxel pattern analysis, this study examined the coding of color with both a simple form feature (orientation) and a mid-level form feature (curvature) in human early visual areas V1 to V4, posterior and central color regions, and shape areas in ventral and lateral occipito-temporal cortex. With the exception of the central color region (which showed color but not form decoding), successful color and form decoding was found in all other regions examined, even for color and shape regions showing univariate sensitivity to one feature. That said, all regions exhibited significant feature decoding biases, with decoding from color and shape regions largely consistent with their univariate preferences. Color and form are thus represented in neither a completely distributed nor a completely modular manner, but a biased distributed manner. Interestingly, coding of one feature in a brain region was always tolerant to changes in the other feature, indicating relative independence of color and form coding throughout the ventral visual cortex. Although evidence for interactive coding of color and form also existed, the effect was weak and only existed for color and orientation conjunctions in early visual cortex. No evidence for interactive coding of color and curvature was found. The predominant relationship between color and form coding in the human brain appears to be one of anatomical coexistence (in a biased distributed manner), but representational independence.

Visual Perceptual Learning of Feature Conjunctions Leverages Non-linear Mixed Selectivity

“…Mixed selectivity neurons explicitly provide task-relevant information in conjunction learning tasks. They have been described in a number of brain areas across species, including in visual cortex (31)(32)(33)(34)(35)(36)(37)(38)(39). Mixed selectivity neurons are thought to improve decoding of information in downstream areas by increasing the dimensionality of the representational space, thus enabling additional hyperplanes for linear decoders operating in population space that can aid difficult and even non-linear discrimination problems (40)(41)(42).…”

Joint representation of color and form in convolutional neural networks: A stimulus-rich network perspective

“…Indeed, although initial evidence from visual search [ 3 ] and neuropsychology studies [ 45 ] suggests that color and form might be initially encoded independently, and only combined in a late binding operation, other strands of evidence suggest that color and form might be encoded in an interactive manner early on during processing [ 46 ]. For example [ 29 ], and our own work [ 47 ] have shown that nonlinear tuning for color/orientation combinations might emerge as early as V1, V2, V3, and V4. Consistent with the present observation, interactive coding of color and form has also recently been observed in the color selective neurons of macaque color patches [ 28 ].…”

Understanding transformation tolerant visual object representations in the human brain and convolutional neural networks

“…At the population level, tolerance is seen as the ability of a group of neurons in macaque IT or fMRI voxels in human occipito-temporal cortex (OTC) to decode object identity across nonidentity feature changes (e.g., testing if a linear classifier trained to differentiate objects at one position is successful at doing so at an untrained position; Hung et al, 2005;Rust & DiCarlo, 2010;Cichy et al, 2011;Taylor & Xu, 2022; see also Ward et al, 2018;Mocz et al, 2021). Neuronal recording and simulation further show that rank-order preservation at the single neuron level and cross-decoding success at the population level are closely correlated .…”

“…At the single neuron level, tolerance is reflected in macaque IT neurons’ ability to preserve the rank order of the neuronal firing rate to different objects across a nonidentity feature change even when the absolute responses rescale with the change (Schwartz et al, 1983; Tovee et al, 1994; Ito et al, 1995; DiCarlo & Manusell, 2003; Brincat & Connor, 2004; DiCarlo and Cox, 2007; Li et al, 2009; Murty & Arun, 2017; see Figure 1a and 1b for a schematic illustration). At the population level, tolerance is seen as the ability of a group of neurons in macaque IT or fMRI voxels in human occipito-temporal cortex (OTC) to decode object identity across nonidentity feature changes (e.g., testing if a linear classifier trained to differentiate objects at one position is successful at doing so at an untrained position; Hung et al, 2005; Rust & DiCarlo, 2010; Cichy et al, 2011; Vaziri-Pashkam & Xu, 2019; Vaziri-Pashkam et al, 2019; Taylor & Xu, 2022; see also Ward et al, 2018; Mocz et al, 2021). Neuronal recording and simulation further show that rank-order preservation at the single neuron level and cross-decoding success at the population level are closely correlated (Li et al, 2009).…”