We compared the ability of psychophysical observers and single cortical neurons to discriminate weak motion signals in a stochastic visual display. All data were obtained from rhesus monkeys trained to perform a direction discrimination task near psychophysical threshold. The conditions for such a comparison were ideal in that both psychophysical and physiological data were obtained in the same animals, on the same sets of trials, and using the same visual display. In addition, the psychophysical task was tailored in each experiment to the physiological properties of the neuron under study; the visual display was matched to each neuron's preference for size, speed, and direction of motion. Under these conditions, the sensitivity of most MT neurons was very similar to the psychophysical sensitivity of the animal observers. In fact, the responses of single neurons typically provided a satisfactory account of both absolute psychophysical threshold and the shape of the psychometric function relating performance to the strength of the motion signal. Thus, psychophysical decisions in our task are likely to be based upon a relatively small number of neural signals. These signals could be carried by a small number of neurons if the responses of the pooled neurons are statistically independent. Alternatively, the signals may be carried by a much larger pool of neurons if their responses are partially intercorrelated.

We have previously documented the exquisite motion sensitivity of neurons in extrastriate area MT by studying the relationship between their responses and the direction and strength of visual motion signals delivered to their receptive fields. These results suggested that MT neurons might provide the signals supporting behavioral choice in visual discrimination tasks. To approach this question from another direction, we have now studied the relationship between the discharge of MT neurons and behavioral choice, independently of the effects of visual stimulation. We found that trial-to-trial variability in neuronal signals was correlated with the choices the monkey made. Therefore, when a directionally selective neuron in area MT fires more vigorously, the monkey is more likely to make a decision in favor of the preferred direction of the cell. The magnitude of the relationship was modest, on average, but was highly significant across a sample of 299 cells from four monkeys. The relationship was present for all stimuli (including those without a net motion signal), and for all but the weakest responses. The relationship was reduced or eliminated when the demands of the task were changed so that the directional signal carried by the cell was less informative. The relationship was evident within 50 ms of response onset, and persisted throughout the stimulus presentation. On average, neurons that were more sensitive to weak motion signals had a stronger relationship to behavior than those that were less sensitive. These observations are consistent with the idea that neuronal signals in MT are used by the monkey to determine the direction of stimulus motion. The modest relationship between behavioral choice and the discharge of any one neuron, and the prevalence of the relationship across the population, make it likely that signals from many neurons are pooled to form the data on which behavioral choices are based.

We have documented previously a close relationship between neuronal activity in the middle temporal visual area (MT or V5) and behavioral judgments of motion (Newsome et al., 1989; Salzman et al., 1990; Britten et al., 1992; Britten et al., 1996). We have now used numerical simulations to try to understand how neural signals in area MT support psychophysical decisions. We developed a model that pools neuronal responses drawn from our physiological data set and compares average responses in different pools to produce psychophysical decisions. The structure of the model allows us to assess the relationship between "neuronal" input signals and simulated psychophysical performance using the same methods we have applied to real experimental data. We sought to reconcile three experimental observations: psychophysical performance (threshold sensitivity to motion stimuli embedded in noise), a trial-by-trial covariation between the neural response and the monkey's choices, and a modest correlation between pairs of MT neurons in their variable responses to identical visual stimuli. Our results can be most accurately simulated if psychophysical decisions are based on pools of at least 100 weakly correlated sensory neurons. The neurons composing the pools must include a broader range of sensitivities than we encountered in our MT recordings, presumably because of the inclusion of neurons whose optimal stimulus is different from the one being discriminated. Central sources of noise degrade the signal-to-noise ratio of the pooled signal, but this degradation is relatively small compared with the noise typically carried by single cortical neurons. This suggests that our monkeys base near-threshold psychophysical judgments on signals carried by populations of weakly interacting neurons; these populations include many neurons that are not tuned optimally for the particular stimuli being discriminated.

The relationship between neuronal activity and psychophysical judgement has long been of interest to students of sensory processing. Previous analyses of this problem have compared the performance of human or animal observers in detection or discrimination tasks with the signals carried by individual neurons, but have been hampered because neuronal and perceptual data were not obtained at the same time and under the same conditions. We have now measured the performance of monkeys and of visual cortical neurons while the animals performed a psychophysical task well matched to the properties of the neurons under study. Here we report that the reliability and sensitivity of most neurons on this task equalled or exceeded that of the monkeys. We therefore suggest that under our conditions, psychophysical judgements could be based on the activity of a relatively small number of neurons.