There is little research on how resident perceptions of neighborhood unsafety develop over time and how changes in these perceptions relate to decreasing crime rates. The present study analyzes and explains trends in perceived neighborhood unsafety within the Dutch city of Rotterdam, based on survey and register data collected in the years 2003–2017 (N = 148.344, 62 neighborhoods). In addition to crime, we also assess to what extent (changes in) the economic status, level of ethnic heterogeneity, degree of residential mobility, and amount of disorder in the neighborhood play a role in how safe or unsafe inhabitants have felt in a 15-year period. We find that unsafety levels steadily declined in the years up to 2007. This decrease was best explained by changes regarding the economic status, victimization rates and disorder level of neighborhoods. After a sudden increase in feelings of unsafety between 2007 and 2008, explained by the shift towards using more self-administrated questionnaires, fear levels stabilized during the remaining years (2008–2017) although recorded crime levels continued to decrease in this period.