There are many health benefits of regular physical activity and improving physical fitness levels can reduce the risk of chronic disease. Accumulating evidence suggests the neighborhood built environment is important for supporting physical activity; however, few studies have investigated the contribution of the neighborhood built environment to fitness levels. We examined the associations between objectively-determined and self-reported neighborhood walkability and overall and specific components of perceived health-related fitness (cardiorespiratory, muscular strength, and flexibility) in a random sample of 592 adults from two areas of Calgary (Canada). Participants provided complete data to an online questionnaire capturing perceived cardiorespiratory fitness (CRF), muscular strength (MST), flexibility, moderate-to-vigorous intensity physical activity (MVPA), resistance training, and sociodemographic characteristics. The questionnaire also captured participant’s perceptions of their neighborhood’s walkability (Physical Activity Neighborhood Environment Scale; PANES) and the physical activity supportiveness of neighborhood parks (Park Perceptions Index; PPI). Objectively-measured neighborhood walkability was estimated using Walk Score®. The average (SD) age of participants was 46.6 (14.8) years and 67.2% were female. Participants, on average, participated in at least 30-minutes of MVPA on 3.4 (2.1) days/week and undertook resistance training 2.0 (1.8) days/week. Adjusting for covariates, Walk Score® was not associated with any fitness outcomes. Adjusting for covariates, the PANES index was positively associated (p < 0.05) with CRF, MST, flexibility, and overall fitness and the PPI was positively associated (p < 0.05) with all fitness outcomes except MST. Our findings provide novel preliminary evidence suggesting the neighborhood built environment may be important for supporting higher health-related fitness levels in adults.