Because of high population density and rapid urbanization, different human life stages have distinct growth experiences, leading to different lifestyles and age-spatial separation in the same neighborhood environment, particularly in smaller Chinese cities. The relationship of environment to physical activity may differ from western or larger Chinese cities. This study examined the associations of walking duration to the neighborhood environment and other factors, and explored the nuances of walking behavior for different life stages of adults in a smaller Chinese city, Yuncheng. An interviewer-administered questionnaire survey (n = 173) and face-to-face interviews (n = 19) were conducted in August 2017. Descriptive analysis and multiple linear regression were performed to describe walking motivations, sociodemographic characteristics, neighborhood environments, and their impacts on walking duration across three life stages. The quantitative findings were followed by interviews to validate and interpret them. Our results showed no positive associations of land-use mix (LUM) and residential density on walking duration, and even inverse associations of LUM-recreation and LUM-education for specific life stages were identified. Younger people's walking behavior was more related to consumption amenities distinct from those of older people. Our findings suggest that using walkable neighborhood policies (e.g., high residential density and land-use mix) to increase physical activity might be ineffective in smaller Chinese cities.