Age-related hearing loss (ARHL) is a public health problem that has been associated with negative health outcomes ranging from increased frailty to an elevated risk of developing dementia. Significant gaps remain in our knowledge of the underlying central neural mechanisms, especially those related to the efferent auditory pathways. Thus, the aim of this study was to quantify and compare age-related alterations in the cholinergic olivocochlear efferent auditory neurons. We assessed, in young-adult and aged CBA mice, the number of cholinergic olivocochlear neurons, auditory brainstem response (ABR) thresholds in silence and in presence of background noise, and the expression of excitatory and inhibitory proteins in the ventral nucleus of the trapezoid body (VNTB) and in the lateral superior olive (LSO). In association with aging, we found a significant decrease in the number of medial olivocochlear (MOC) cholinergic neurons together with changes in the ratio of excitatory and inhibitory proteins in the VNTB. Furthermore, in old mice we identified a correlation between the number of MOC neurons and ABR thresholds in the presence of background noise. In contrast, the alterations observed in the lateral olivocochlear (LOC) system were less significant. The decrease in the number of LOC cells associated with aging was 2.7-fold lower than in MOC and in the absence of changes in the expression of excitatory and inhibitory proteins in the LSO. These differences suggest that aging alters the medial and lateral olivocochlear efferent pathways in a differential manner and that the changes observed may account for some of the symptoms seen in ARHL.