Age-related hearing loss is a global health problem of increasing importance. While the role of peripheral hearing loss is well understood and treatments are available, central hearing loss, the ability of the brain to make sense of sound, is much less well understood and no treatments are available. We report on age-related alterations in the auditory brain stem which compromise a listener's ability to isolate a sound from competing background noises, for example in a crowded restaurant. Sound localization depends on extreme temporal precision on the order of microseconds, and the sound localization pathway shows several specializations towards temporal precision. The pathway from the cochlear nucleus to the medial nucleus of the trapezoid body (MNTB) is heavily myelinated and terminates in the calyx of Held. Using auditory brain stem response measurements (ABRs), we found that the physiological properties of MNTB changes with age. The mechanism is that in older animals, MNTB afferents demyelinate to various degrees, resulting in larger variability in the timing of responses. Myelin is produced by oligodendrocytes, and we found that fewer mature, but more precursor and immature oligodendrocytes are present in MNTB of aged animals, suggesting that the demyelination is an age-related deficit in oligodendrocyte maturation.