Presbycusis or age-related hearing loss is a prevalent condition in the elderly population, which affects oral communication, especially in background noise, and has been associated with social isolation, depression, and cognitive decline. However, the mechanisms that relate hearing loss with cognition are complex and still elusive. Importantly, recent studies show that the use of hearing aids in presbycusis, which is its standard management, can induce neuroplasticity and modify performance in cognitive tests. As the majority of the previous studies on audition and cognition obtained their results from a mixed sample of subjects, including presbycusis individuals fitted and not fitted with hearing aids, here, we revisited the associations between hearing loss and cognition in a controlled sample of unaided presbycusis. We performed a cross-sectional study in 116 non-demented Chilean volunteers aged ≥65 years from the Auditory and Dementia study cohort. Specifically, we explored associations between bilateral sensorineural hearing loss, suprathreshold auditory brain stem responses, auditory processing (AP), and cognition with a comprehensive neuropsychological examination. The AP assessment included speech perception in noise (SIN), dichotic listening (dichotic digits and staggered spondaic words), and temporal processing [frequency pattern (FP) and gap-in-noise detection]. The neuropsychological evaluations included attention, memory, language, processing speed, executive function, and visuospatial abilities. We performed an exploratory factor analysis that yielded four composite factors, namely, hearing loss, auditory nerve, midbrain, and cognition. These four factors were used for generalized multiple linear regression models. We found significant models showing that hearing loss is associated with bilateral SIN performance, while dichotic listening was associated with cognition. We concluded that the comprehension of the auditory message in unaided presbycusis is a complex process that relies on audition and cognition. In unaided presbycusis with mild hearing loss (<40 dB HL), speech perception of monosyllabic words in background noise is associated with hearing levels, while cognition is associated with dichotic listening and FP.