Buildings’ built environment has been linked to their occupants’ health. It remains unclear whether those elements that predisposed its residents to poor general health before the two SARS pandemics also put residents at risk of contracting COVID-19 during early outbreaks. Relevant research to uncover the associations is essential, but there lacks a systematic examination of the relative contributions of different elements in one’s built environment and other non-environmental factors, singly or jointly. Hence, the current study developed a deep-learning approach with multiple input channels to capture the hierarchical relationships among an individual’s socioecology’s demographical, medical, behavioral, psychosocial, and built-environment levels. Our findings supported that 1) deep-learning models whose inputs were structured according to the hierarchy of one’s socioecology outperformed plain models with one-layered input in predicting one’s general health outcomes, with the model whose hierarchically structured input layers included one’s built environment performed best; 2) built-environment features were more important to general health compared to features of one’s sociodemographic and their health-related quality of life, behaviors, and service utilization; 3) a composite score representing built-environment features’ statistical importance to general health significantly predicted building-level COVID-19 case counts; and 4) building configurations derived from the expert-augmented learning of granular built-environment features that were of high importance to the general health were also linked to building-level COVID-19 case counts of external samples. Specific built environments put residents at risk for poor general health and COVID-19 infections. Our machine-learning approach can benefit future quantitative research on sick buildings, health surveillance, and housing design.HighlightsThe current modeling approaches for COVID-19 transmission at early spread are limited due to uncertainty and rare events.Socio-ecological structure (SES) can organize variables from different hierarchies of a total environment.TensorFlow-based deep learning with recurrent and convolutional neural networks is developed to explain general health with SES-organized variables.Among SES factors, built environments have a greater association with general health.Built-environment risks on individual general health associated with early-spread COVID-19 infections in residential buildings.