This article seeks to explain the emergence of a local protest movement against an asylum seeker center (“asielzoekerscentrum” or AZC in Dutch) in the Beverwaard neighborhood in the city of Rotterdam (NL). Based on the contradictory evidence found on the connection between material conditions and anti-immigration mobilization, this article seeks to expand the literature by viewing materiality through the lens of storytelling. Through a qualitative analysis of twenty-eight interviews with inhabitants, local politicians, civil servants, and social workers in 2017, this article illustrates how storytelling about territorial stigmatization and material deprivation played a role in mobilizing inhabitants against the establishment of an AZC in their area two years prior, in 2015. Overall, this article argues that it is not materiality per se but inhabitants’ reading of materiality through practices of storytelling that informed collective mobilization against the arrival of an AZC in a relatively deprived urban community.