The ongoing pandemic is laying bare dramatic differences in the spread of COVID-19 across seemingly similar urban environments. Identifying the urban determinants that underlie these differences is an open research question, which can contribute to more epidemiologically resilient cities, optimized testing and detection strategies, and effective immunization efforts. Here, we perform a computational analysis of COVID-19 spread in three cities of similar size in New York State (Colonie, New Rochelle, and Utica) aiming to isolate urban determinants of infections and deaths. We develop detailed digital representations of the cities and simulate COVID-19 spread using a complex agent-based model, taking into account differences in spatial layout, mobility, demographics, and occupational structure of the population. By critically comparing pandemic outcomes across the three cities under equivalent initial conditions, we provide compelling evidence in favor of the central role of hospitals. Specifically, with highly efficacious testing and detection, the number and capacity of hospitals, as well as the extent of vaccination of hospital employees are key determinants of COVID-19 spread. The modulating role of these determinants is reduced at lower efficacy of testing and detection, so that the pandemic outcome becomes equivalent across the three cities. Electronic supplementary material The online version of this article (10.1007/s11524-022-00623-9) contains supplementary material, which is available to authorized users.