Listening in a noisy environment (e.g., speech in noise) relies on the fundamental ability to extract coherence from the variable sensory input. This allows the detection of active sound sources and their segregation from the rest of the scene (figure-ground segregation). Peripheral and central causes of age-related decline of listening in noise were assessed by a tone-cloud-based figure detection task. In two conditions differing in the amount of noise, figure detection performance was equalized between young, normal-hearing, and hearing-impaired elderly listeners by adapting the stimulation separately to the abilities of each person. Based on behavioral measures and event-related brain potentials (ERP), in the absence of cognitive deficits, aging alone does not appear to significantly deteriorate the ability to detect sound sources in noise, although ERPs show delayed perceptual processes and some expected deterioration in attention and/or executive functions. However, even mild hearing impairment substantially reduces the ability to segregate individual sound sources within a complex auditory scene, and susceptibility to masking noise increases together with the severity of the hearing deficit.