Built environment characteristics such as walkability, land use diversity, infrastructure accessibility and attractiveness may support or hinder the elderly’s leisure activities, which in turn affects their health. Promoting the elderly’s leisure activities through the creation of a positive built environment is of great relevance to healthy aging. In the context of the continuous increasing of aging in China, promoting leisure activities for the elderly through improving the built environment has become an essential issue in urban geography and urban planning. Based on the questionnaire survey data of the elderly in Hefei City, a multilevel ordered probit regression model was used to investigate the mechanism of the multi-scale built environment on leisure activities of the elderly. The results show that: (1) more than 60% of the elderly can carry out leisure activities more than seven times a week, more than 50% of the elderly have a duration of fewer than 30 min for each leisure activity, and there are significant spatial differences in the frequency and duration of their trips at multiple scales in city, community and residential district. (2) Residential quality and community-level land use mixture, the density of leisure facilities, proximity to high-level urban roads, community security, living in the old city, and individual characteristic variables such as age, education, and satisfaction with neighborhood interaction positively contribute to the leisure activities of the elderly. In contrast, the community activity participation and the location close to expressways and railway lines have a significantly negative impact on the leisure activities of the elderly. (3) The mechanism of interactions between multi-scale built environments on the leisure activities of the elderly is mainly summarized as the transmission effect and substitution effect. The transmission effect shows that the differences in the community-level built environment are primarily caused by the differences in the city-level built environment. In contrast, the substitution effect shows that the multi-scale built environment such as residential districts, communities, and cities jointly affect the leisure activities of the elderly. Based on the mechanism of the built environment at different scales, this study can provide theoretical references and planning implications to improve the built environment through planning means such as enhancing the walkability of streets, the accessibility of facilities and the scale of greenery in order to promote active leisure activities and improve the health of the elderly.