Background: Neighborhood walkability achieved through land-use and transport planning is an important determinant of physical activity, especially for older adults who spend more time in their neighborhoods than do other ages. This three-year study aimed to examine the longitudinal association between the change in the step count of older adults and the built environment (BE) around their homes in Yokohama, Japan. Methods: We analyzed pedometer data in March 2016 and March 2019 that was acquired from 21 557 older adults aged 65–79 years at baseline, who lived in 758 neighborhoods in Yokohama city who participated in the Yokohama Walking Point Program (YWPP). Neighborhoods were classified into quartiles for each of the six built environmental variables (population density, intersection density, the proportion of commercial land use, normalized difference vegetation index (NDVI), the average distance to the nearest railway station, and average distance to the nearest bus stop), which were selected based on previous studies. Using multilevel regression analysis, we examined the connection between the BE variables, baseline step count, and the change in step count during the three years. Results: Higher population density, lower intersection density, and the second shortest quartile of the average distance to the nearest railway station were associated with a higher baseline step count. A lower intersection density and shorter average distance to the nearest railway station were associated with a smaller decline. The lowest quartile of population density was reversely associated. Additionally, female and older groups recorded a lower baseline step count and a larger decline. A higher baseline step count was associated with a larger decline, as well. Conclusions: The neighborhood BEs (i.e., population density, intersection density, and average distance to the nearest railway station) of older adults were not only associated with their step count at a certain time point, but also widened the disparity of the step count over the three years. Urban planners and designers should place emphasis on interdisciplinary collaboration with health promotion professionals in order to create compact cities centered around railway stations that are protected from traffic, so older adults feel safe.