The residualization of public rented housing is a prevalent phenomenon throughout Europe, and strongly present in the small and strongly means-tested social housing sector in Norway. In this article, we discuss the contested geographical dimension of residualization. Scientific studies of the geographical and locational aspects of social housing are scare in Norway and modest internationally. Based on qualitative interviews with representatives of social housing administrators in the fifteen largest urban municipalities in Norway, this paper contributes to the literature by exploring how these social housing bureaucrats perceive, reflect on, and respond to, questions related to the spatial localization of residual social housing. Does it matter where social housing is located? What are the consequences of the geography of social housing for tenants, their neighbours, and the wider socio-spatial development of cities? These are questions pondered in the interviews. In our qualitative analysis, we identify three broad themes. First, the theme of the internal social milieu – inclusive communities versus neighbour complaints and conflicts in the public housing projects. Second, the theme of neighbourhood effects; how concentrated poverty is influencing the local community in general and the upbringing of children in particular. Third, the theme of response from external neighbours and communities, in the form of either predominantly exclusive strategies (NIMBYism – Not in My Backyard), but also less prevalent inclusive strategies like (PHIMBYism – Public Housing In My Backyard).