The health implications of environmental noise, especially cardiovascular effects, have been studied intensively. Research on associations between noise and mental health, however, has shown contradictory results. The present study examined associations between individual levels of noise annoyance due to noise from various sources in the living environment and mental health of adults in Germany. It evaluated whether these associations persisted after adjusting for potential covariates. Data were obtained from the cross-sectional “German Health Update” study 2012 (GEDA 2012), a national health interview survey among adults in Germany conducted by the Robert Koch Institute (n = 19,294). Noise annoyance questions referred to overall noise and that from road traffic, neighbours, and air traffic. Mental health was measured with the five-item Mental Health Inventory. Bivariate analysis showed associations between high levels of noise annoyance and impaired mental health for all noise sources except air traffic. After adjusting for covariates (sociodemographic factors, chronic disease, and social support), both men and women who reported high overall noise annoyance showed more than doubled odds of impaired mental health compared to those who were not annoyed. The odds of impaired mental health in the highest noise annoyance category from road traffic and neighbours were also significantly increased. These findings indicate that high noise annoyance is associated with impaired mental health and that this association can vary with the source of environmental noise. Further research on covariates of this association is necessary. Particularly, longitudinal data are required to establish the direction of associations and to address questions of causality.