Objective Perceived neighborhood environment (NE) is considered a determinant of daily physical activity (PA). However, evidence concerning differences among types of PA, gender, and age is limited. The study purpose was to clarify the association between NE and walking time (WT) or habitual exercise (HE) across gender, age, and employment status in a community-dwelling population of Japan. Methods A questionnaire mail survey with a stratified random 7,515 sampling was conducted in a northeast city in January 2007. Multiple logistic analysis was conducted to examine the associations between seven NE indices and WT or HE across gender, age, and employment status: 20-39 (young-employed), 40-59 (middle-employed), and 60-79 (old-employed or old-unemployed) after adjustment for age and means of transportation. Results A total of 3,806 residents (52.4 % females) completed the survey. Traffic and crime safety in oldunemployed males and proximity to service facilities and traffic and crime safety in middle-employed females were significantly associated with a low risk of insufficient WT. Proximity to service facilities in old-employed males, number of service facilities, places for walking, and good view in middle-employed females, and density of dwelling and proximity to service facilities in old-unemployed females were significantly associated with a low risk of non-HE. Conclusions The association between NE and WT or HE differed across type of PA, gender, age, and employment status, and was observed mainly in middle-and old-aged females. The middle-and old-aged female residents' PA possibly were more influenced by their NE, and NE would contribute to promote active living.