Poverty and its associated factors put people at risk for depression. The aims of this study were to describe the prevalence of depressive symptoms (DS) of primary caregivers and socioemotional development (SED) delays of young children in poor rural areas of China, and to explore the association between them. Cross-sectional data of 2,664 children aged 3 to 35 months and their primary caregivers were used for analysis. Characteristics of the child, caregiver, and family were collected through face-to-face caregiver interviews. DS were assessed by the Zung Self-Rating Depression Scale (W.W. Zung, 1965, as cited in World Health Organization, ), and SED was evaluated by the Ages and Stage Questionnaires: Social-Emotional (J. Squires, D. Bricker, & L. Potter, 1997). The χ test, stratification analysis, and logistic regression analyses were used to explore the association. Among the caregivers, 40.3% (95% confidence interval [CI] [38.4, 42.1]), reported DS. Caregivers who were male, older and ethnic minorities as well as had a low level of education, a low family income, or more children were more likely to have DS. Of the children, 24.4% (95% CI [22.8, 26.0]) were recognized with SED delays. Older children displayed more delays than did younger children, but no significant differences between males and females were found. SED delays were significantly associated with mother outmigrating, male caregivers, older age, ethnic minorities, and low education or families with a single parent, low-income, and having more children. Caregivers having DS, odds ratio (OR) = 2.40, 95% CI [1.99, 2.88], was a significant predictor of increased odds of SED delays; other factors were single-parent family, OR = 1.99, 95% CI [1.37, 2.89], inadequate care, OR = 1.69, 95% CI [1.30, 2.21], physical punishment, OR = 1.61, 95% CI [1.33, 1.95], ethnic minorities, OR = 1.41, 95% CI [1.17, 1.71], and child age in months, OR = 1.03, 95% CI [1.02, 1.04], according to the logistic regression analysis. DS are prevalent among caregivers with young children in poor rural areas. Interventions to improve the mental health of caregivers and their parenting behaviors are needed to improve children's SED.