The mammalian auditory system is the temporally most precise sensory modality: To localize low-frequency sounds in space, the binaural system can resolve time differences between the ears with microsecond precision. In contrast, the binaural system appears sluggish in tracking changing interaural time differences as they arise from a low-frequency sound source moving along the horizontal plane. For a combined psychophysical and electrophysiological approach, we created a binaural stimulus, called "Phasewarp," that can transmit rapid changes in interaural timing. Using this stimulus, the binaural performance in humans is significantly better than reported previously and comparable with the monaural performance revealed with amplitude-modulated stimuli. Parallel, electrophysiological recordings of binaural brainstem neurons in the gerbil show fast temporal processing of monaural and different types of binaural modulations. In a refined electrophysiological approach that was matched to the psychophysics, the seemingly faster binaural processing of the Phasewarp was confirmed. The current data provide both psychophysical and physiological evidence against a general, hard-wired binaural sluggishness and reconcile previous contradictions of electrophysiological and psychophysical estimates of temporal binaural performance.