Against the backdrop of accelerating population aging and scarce elderly care resources and facilities in China, the issue of the sense of safety among the elderly has received widespread attention. This article is based on data from the China Labor Force Dynamics Survey (CLDS) in 2016 and takes three major urban agglomerations as examples to examine the impact mechanisms of urban built environments on elderly people’s sense of safety. The results indicate that the characteristics of the urban built environment, the social environment, and individual health affect the safety perceptions of the elderly. Among them, urbanization rate, hospital facilities, population density, greening rate, air quality, and frequency of dining out have significant impacts on elderly people’s sense of safety. Simultaneously, good daily exercise and mental health status can significantly improve elderly people’s sense of safety. This article summarizes the existing problems of aging-friendly spaces and facilities in three major urban agglomerations, proposes planning strategies to enhance elderly people’s sense of safety, and provides a useful reference for urban aging-friendly transformations and an elderly-friendly society.