The built environment can contribute to the spread of the novel coronavirus disease (COVID-19) by facilitating human mobility and social contacts between infected and uninfected individuals. However, mobility data capturing detailed interpersonal transmission at a large scale are not available. In this study, we aimed to objectively assess the influence of key built environment factors, which create spaces for activities—“inferred activity” rather than “actually observed activity”—on the spread of COVID-19 across townships in China at its initial stage through a random forest approach. Taking data for 2994 township-level administrative units, the spread is measured by two indicators: the ratio of cumulative infection cases (RCIC), and the coefficient of variation of infection cases (CVIC) that reflects the policy effect in the initial stage of the spread. Accordingly, we selected 19 explanatory variables covering built environment factors (urban facilities, land use, and transportation infrastructure), the level of nighttime activities, and the inter-city population flow (from Hubei Province). We investigated the spatial agglomerations based on an analysis of bivariate local indicators of spatial association between RCIC and CVIC. We found spatial agglomeration (or positive spatial autocorrelations) of RCIC and CVIC in about 20% of all townships under study. The density of convenience shops, supermarkets and shopping malls (DoCSS), and the inter-city population flow (from Hubei Province) are the two most important variables to explain RCIC, while the population flow is the most important factor in measuring policy effects (CVIC). When the DoCSS gets to 21/km2, the density of comprehensive hospitals to 0.7/km2, the density of road intersections to 72/km2, and the density of gyms and sports centers to 2/km2, their impacts on RCIC reach their maximum and remain constant with further increases in the density values. Stricter policy measures should be taken at townships with a density of colleges and universities higher than 0.5/km2 or a density of comprehensive hospitals higher than 0.25/km2 in order to effectively control the spread of COVID-19.