Depression amongst the elderly population is a worldwide public health problem, especially in China. Affected by the urban–rural dual structure, depressive symptoms of the elderly in urban and rural areas are significantly different. In order to compare depressive symptoms and its influencing factors among the elderly in urban and rural areas, we used the data from the fourth wave of the China Health and Retirement Longitudinal Study (CHARLS). A total of 7690 participants at age 60 or older were included in this study. The results showed that there was a significant difference in the prevalence estimate of depression between urban and rural elderly (χ2 = 10.9.76, p < 0.001). The prevalence of depression among rural elderly was significantly higher than that of urban elderly (OR -unadjusted = 1.88, 95%CI: 1.67 to 2.12). After adjusting for gender, age, marital status, education level, minorities, religious belief, self-reported health, duration of sleep, life satisfaction, chronic disease, social activities and having income or not, the prevalence of depression in rural elderly is 1.52 times (OR = 1.52, 95% CI: 1.32 to 1.76) than that of urban elderly. Gender, education level, self-reported health, duration of sleep, chronic diseases were associated with depression in both urban and rural areas. In addition, social activities were connected with depression in urban areas, while minorities, marital status and having income or not were influencing factors of depression among the rural elderly. The interaction analysis showed that the interaction between marital status, social activities and urban and rural sources was statistically significant (divorced: coefficient was 1.567, p < 0.05; social activities: coefficient was 0.340, p < 0.05), while gender, education level, minorities, self-reported health, duration of sleep, life satisfaction, chronic disease, social activities having income or not and urban and rural sources have no interaction (p > 0.05). Thus, it is necessary to propose targeted and precise intervention strategies to prevent depression after accurately identifying the factors’ effects.