Humans and animals constantly face challenging acoustic environments such as various background noises restricting the detection, discrimination and identification of behaviorally salient sounds. Here, we disentangled the role of temporal envelope tracking on the decrease in neuronal and behavioral discrimination between communication sounds in situations of acoustic degradations. We simulated responses of auditory nerve fibers and recorded neuronal activity in cochlear nucleus, inferior colliculus, thalamus and auditory cortex in anesthetized guinea-pigs. Furthermore, a Go/No-Go sound discrimination task involving two of the guinea-pig whistles was performed on mice in silence and noise. For all conditions, we found that auditory neurons better track the slow amplitude modulations (<20 Hz) of the stimulus envelopes than the faster ones. In addition, the decrease in neuronal and behavioral discrimination performance in noise can be explained by an increased similarity of the vocalization envelopes in the low frequency range (<20 Hz). Together, these results suggest that slow envelope tracking is a general property of auditory neurons, and any difference between the slow envelopes of natural stimuli allows coping with degraded conditions.