Previous studies have documented numerous health benefits of conducting regular physical activity among older adults. The built environment is believed to be a key factor that can hinder or facilitate daily physical activity, such as walking and exercising. However, most empirical studies focusing on environment-physical activity associations exhibited residential self-selection bias with cross-sectional research design, engendering doubts about the impact of built environment on physical activity. To reduce this bias, we assessed physical activity behaviors of 720 Hong Kong older adults (≥65 years) residing in 24 public housing estates. The Hong Kong public housing scheme currently provides affordable rental flats for 2.1 million people or approximate 30% of total population. The applicants were allocated to one of 179 housing estates largely by family size and flat availability. Built environment characteristics were measured following the ‘5Ds’ principle: (street network) design, (land-use) diversity, density, distance to transit, and destination accessibility. Multilevel mixed models were used to explore the associations between the built environment and the different domains of physical activity (transportation walking, recreational walking, and recreational moderate-to-vigorous physical activity (MVPA) while controlling for potential estate-level socioeconomic and individual confounders. We found that transportation walking was positively associated with the number of bus stops and the presence of Mass Transit Railway (MTR) stations. Recreational MVPA was positively related to the number of recreational facilities. However, land-use mix was negatively related to transportation walking, recreational walking, and recreational MVPA. The findings of this study support a threshold effect in the environment-physical activity associations. Furthermore, large-scale public housing schemes involving random or semi-random residence assignment in many cities may provide opportunities to explore built environments and physical activity behavior, with the potential to overcome residential self-selection bias.