The built environment has been linked to physical activity (PA) behaviors, yet there is limited knowledge of this association among lower-income midlife and older adults who are insufficiently active. The present cross-sectional study utilized baseline data collected between October 2017 and November 2019 from a clustered randomized controlled trial to determine how built environment attributes were associated with PA behaviors among midlife and older adults (n = 255) residing in or near affordable housing sites (n = 10). At each site, perceptions of the built environment were collected and scored at the participant level via the abbreviated Neighborhood Environment Walkability Survey (NEWS-A), while objective built environment attributes were measured and scored by trained research staff using the Physical Activity Resource Assessment (PARA). Multiple PA behaviors—walking, total PA, and moderate-to-vigorous PA (MVPA) (min/wk)—were measured using the validated Community Healthy Activities Model Program for Seniors (CHAMPS) questionnaire. Adjusted linear regression models examined associations between NEWS-A measures and PA behaviors, and site-level correlations between PARA measures and PA behaviors were examined using Spearman’s rank correlations. At the participant level, adjusted models revealed that a one point increase in the NEWS-A aesthetics score was associated with a 57.37 min/wk increase in walking (β = 57.37 [95% CI: 20.84, 93.91], p = 0.002), with a similar association observed for street connectivity and MVPA (β = 24.31 min/wk [95% CI: 3.22, 45.41], p = 0.02). At the site level, MVPA was positively correlated with the quality of the features of local, PA-supportive environmental resources (ρ = 0.82, p = 0.004). Findings indicate that participant- and site-level measures of the built environment may play a role in promoting PA behavior among this demographic and similar populations. Results also suggest that improvements in aesthetic attributes and street connectivity, along with enhancing the quality of local, PA-supportive environmental resources, may be effective strategies for promoting physical activity among lower-income midlife and older adults.