Within a city, gender differences in walking for recreation (WfR) vary significantly across neighbourhoods, although the reasons remain unknown. This cross-sectional study investigated the contribution of the social environment (SE) to explaining such variation, using 2009 data from the How Areas in Brisbane Influence healTh and AcTivity (HABITAT) study, including 7866 residents aged 42–67 years within 200 neighbourhoods in Brisbane, Australia (72.6% response rate). The analytical sample comprised 200 neighbourhoods and 6643 participants (mean 33 per neighbourhood, range 8–99, 95% CI 30.6–35.8). Self-reported weekly minutes of WfR were categorised into 0 and 1–840 mins. The SE was conceptualised through neighbourhood-level perceptions of social cohesion, incivilities and safety from crime. Analyses included multilevel binomial logistic regression with gender as main predictor, adjusting for age, socioeconomic position, residential self-selection and neighbourhood disadvantage. On average, women walked more for recreation than men prior to adjustment for covariates. Gender differences in WfR varied significantly across neighbourhoods, and the magnitude of the variation for women was twice that of men. The SE did not explain neighbourhood differences in the gender–WfR relationship, nor the between-neighbourhood variation in WfR for men or women. Neighbourhood-level factors seem to influence the WfR of men and women differently, with women being more sensitive to their environment, although Brisbane’s SE did not seem such a factor.