The effects of variation from stimulus to stimulus in emotional tone of voice on speech perception were examined through a series of perceptual experiments. Stimuli were recorded from human speakers who produced utterances in tones of voice designed to convey affective information. Then, stimuli varying in talker voice and emotional tone were presented to listeners for perceptual matching and classification. The results showed that both intertalker variation in talker voice and intratalker variation in emotional tone had a negative effect on perceptual performance. The results suggest that sources of variation in the speech signal that affect the spectral/temporal properties of speech (i.e., talker voice, speech rate, emotional tone) may be treated differently than sources of variation that do not affect these properties (i.e., vocal amplitude).