Noise from neighbours has been shown to be one of the most noise annoying sources in Germany, but research on the influencing factors for the annoyance ratings is scarce. Therefore, we investigated whether different personal and contextual (social, physical) factors contribute to neighbour noise annoyance to better understand the neighbour noise annoyance situation. A population-representative survey in four areas in Germany was conducted, with each area further stratified according to their density of agglomeration (inner city, urban outskirt, rural area). Randomly selected residents from each area were invited by mail to participate in the study, either online or via a paper–pencil mode. Noise annoyance was assessed for different noise sources (e.g., neighbourhood, road, railway, aircrafts, different types of industry). In total, 1973 questionnaires were completed. We identified several factors to be predictive of neighbour noise annoyance: satisfaction with the neighbourhood, relationship with neighbours, residential satisfaction, noise sensitivity, and density of agglomeration for people living in the inner city in comparison to rural areas. Particularly, social aspects such as the relationship with neighbours and satisfaction with the neighbourhood have been shown to affect neighbour noise annoyance.