North American downtowns are struggling to recover from the global COVID-19 pandemic. This study aims to investigate the varying rates of recovery experienced by downtown areas in the 66 largest cities of the United States and Canada. Leveraging Location-Based Services data extracted from mobile phone location trajectories, we assess the recovery rates in the 2023 post-pandemic period, juxtaposed against pre-pandemic 2019 levels. We find significant disparities in downtown recovery rates. Economic factors emerge as crucial determinants, where downtowns hosting a concentration of sectors with remote/hybrid work options – such as information, finance, professional services and management – displayed sluggish recovery. Conversely, downtowns with a focus on industries like accommodation, manufacturing, education, retail, construction, entertainment and healthcare exhibited greater resilience post pandemic. Furthermore, higher density, crime rates and education levels were correlated with slower recovery rates, as were harsher weather conditions and longer commuting times. Lower-density and auto-orientated downtowns demonstrated a swift rebound, even surpassing pre-pandemic activity levels. These findings underscore the necessity for tailored policies to bolster the revival of North American downtown areas.