The construction of community elderly care facilities (CECF) is pivotal for promoting healthy aging and “aging in place” for older people. This study focuses on the low utilization rates of community elderly care facilities in the Dongcheng and Xicheng Districts, core areas of Beijing. The explainable machine learning method is used to analyze data across three dimensions: the elderly’s individual attributes, characteristics of the community elderly care station (CECS), and features of the built environment around CECS and subdistrict, to identify the important factors that influence the usage frequency of overall CECS and its different functional spaces, and also the correlation between factors and usage frequency of CECS. It shows that the most important factors are the features of CSCF, including the degree of space acceptance and satisfaction with services provided, which influence the usage frequency of nine functional spaces (R2 ≥ 0.68) and overall (R2 = 0.56). In addition, older people’s individual factors, such as age and physical condition, significantly influence the usage of specific spaces such as rehabilitation therapy rooms and assistive bathing rooms. The influence of built environment characteristics is relatively low, with factors such as the density of bus stations and housing prices within the subdistrict and the mean distance from CECF to the nearest subway stations being more important. These findings provide a reference for the construction of indoor environments, management of service quality, and optimal site selection for future community elderly care facilities.