It is surprising difficult to define where a city center lies, yet its location has a profound effect on a city's structure and function. We examine whether city center typicality points can be consistently located on historical maps such that their centroid identifies a meaningful central location over a 500-year period in Southampton, UK. We compare movements of this city center centroid against changes in the geographical center of the city as defined by its boundary. Southampton's historical maps were georectified with a mean accuracy of 21 m (range 9.9 to 47 m), and 18 to 102 typicality points were identified per map, enough to chart changes in the city center centroid through time. Over nearly 500 years, Southampton's center has moved just 343 m, often corresponding with the key retail attractants of the time, while its population has increased 80-fold, its administrative area 60-fold and its geographical center moved 1985 m. This inertia to change in the city center presents environmental challenges for the present-day, made worse by the geography of Southampton, bounded by the sea, rivers and major roads. Geographical context, coupled with planning decisions in the past that maintain a city center in its historical location, place limits on the current sustainability of a city.