Investing in early childhood development is an effective way to enhance human capital accumulation. Caregivers’ mental health is one of the most important factors influencing children’s development. Previous studies have found that mental health issues in caregivers are widespread all over the world, especially in low- and middle-income countries. In this study, we explored the effects of the “Integrated Program for Early Childhood Development” on the mental health of female caregivers in Southwest China through a randomized intervention trial, with infants aged 5–25 months and their caregivers as the target subjects. The heterogeneity of the effects of different characteristics of the caregivers and the mechanism of the intervention effect were also analyzed. Primary caregivers were provided comprehensive early development interventions for the children in the treatment group via bi-weekly home visiting activities and monthly family group activities. The results showed that the prevalence of depression, anxiety, and stress symptoms among female caregivers in this rural area were 32%, 42%, and 30%, respectively. Whether the child was breastfed, parent’s age, parent’s education level, primary caregiver type, the ratio of the number of months the mother was at home full time to the child’s age, the grandmother’s rearing ability, and the family asset index were the factors influencing the mental health of female caregivers. The intervention significantly increased the proportion of depressive symptoms in 28% of the grandmothers. It significantly reduced the anxiety symptoms of daughters-in-law not from the local town, while the social interactions of both local and non-local daughters-in-law were significantly improved.