This study aimed to compare and analyse the differences in smoking prevalence, and knowledge, attitudes, and factors associated with smoking between the rural and urban elderly population in China. In total, 6,966 participants aged 60 and above were included in this study, which assessed their smoking-related knowledge, attitudes, and perceptions toward tobacco control. The Chi-square test and logistic regression model were used for statistical analysis, and the Fairlie model was used for decomposition analysis. The overall prevalence of smoking was 25.6%; the rate was much higher in men than in women (overall: OR = 26.234; urban: OR = 31.260; rural: OR = 23.889). The rate of correct responses to all questions on smoking problems was significantly higher among the urban elderly than the rural elderly. Further, 64.18% of the participants supported printing photos of the health hazards of smoking on the cover of cigarette packs, and the rural elderly were more supportive of this. Moreover, only 36.52% of the participants supported increasing taxation and retail price of cigarettes; the urban elderly showed more support for this. Rules about smoking at home also played an important role, especially for families where smoking was not allowed at home, but with exceptions to the rule; however, this factor was only meaningful in urban families (urban: OR = 0.117). Through the Fairlie decomposition analysis, gender (-1.62%), age (-2.03%), region (13.68%), knowing about e-cigarettes (5.17%), rules about smoking at home (3.95%), and smoking-related knowledge scores (42.85%) were found to be associated with rural-urban disparities. This study focused on the differences in smoking between urban and rural areas in China. Smoking among the urban elderly was significantly less prevalent compared with the rural population. Factors including education, region, and smoking-related knowledge need to be addressed to reduce the gap between urban and rural health hazards in China.