Background: The health of urban residents plays an important role in increasing the well-being of citizens, and therefore promoting the sustainable development of cities. Studies in Western countries have provided some evidence of the relationship between the urban residents’ perception of the recreational sports (RS), including perception of industry, space, experience, type, and support affecting recreational behaviors. Yet, evidence in China is not sufficient and, particularly, gender difference has not been discussed anywhere. Objectives: The present study aimed to see urban residents’ perception of RS and its effect on recreational sport participation by the residents. Method: The study was conducted in three big cities (Hangzhou, Chengdu, and Shanghai) of China to test the association between the urban residents’ perception of RS and their RS participation, in which gender differences were discussed. After taking their consent, a total of 764 participants (397 male and 367 female) aged from 15 to 65 were approached between April and October, 2018. The participants were given a survey to assess their perception and behavior of RS. T-test and the structural equation modeling were used to analyze gender differences in association between perception and behavior of RS. Results: For females, industry perception (β = 0.17, p = 0.00), space perception (females: β = 0.16, p = 0.00), and support perception (females: β = 0.09, p = 0.03) had a significant positive effect on RS participation. For males, industry perception (males: β = 0.13, p = 0.01) and type perception (males: β = 0.23, p = 0.00) showed a significant positive impact on participation of RS. However, space perception of males (β = −0.12; p = 0.01) and type perception of females (β = −0.11; p = 0.01) are negatively associated with their RS participation. The perception of experience was the most important factor influencing RS behaviors. Significant gender differences have appeared in all five different perception factors. The industry perception has significant and positive effects on both males and females’ RS participation. The type perception has significant positive effects on the male but negative on the female RS, while the space perception is just the opposite. The support perception has significant positive effects on the female RS practice but the influence is not obvious in the case of males. Conclusion: By adopting some measures, perception of RS may improve urban residents’ participation in RS.