This study was to investigate the association of built environment (BE) with physical activity (PA) and physical fitness of residents inside the city wall of Xi’an, one of the most historic cities in China. The cross-sectional study was conducted among 728 adults in this area. BE and PA were measured by Neighborhood Environment Walkability Scale and International Physical Activity Questionnaire, respectively. Body mass index (BMI), body fat percentage, blood pressure (BP), lung capacity, curl up, sit and reach, reaction time, balance, and grip strength were also measured. The results showed, for men, aesthetics was positively associated with total and leisure-time PA, and curl-up tests, respectively. Walking/cycling facilities were positively associated with leisure-time PA. Street connectivity was negatively associated with leisure-time PA and BMI. Residential density was positively correlated with BMI. Access to service was positively associated with lung capacity. Crime safety was negatively correlated with reaction time. For women, residential density was negatively associated with transportation PA and BP. Street connectivity was positively associated with curl-up test. The results suggest some BE attributes are positively related to PA and physical fitness in this population. Creating more PA-supportive BEs is recommended in this historic area along with urban conservation.