Purpose To examine the status of spouses' burdens of caring for breast cancer survivors and explore the relationships between social support, family resilience, breast cancer survivors' individual resilience, and caregiver burden. Methods A cross-sectional study on 315 young and middle-aged breast cancer survivors and their spousal caregivers was conducted at eight comprehensive Southwest China hospitals. The caregivers completed the Chinese Version of the Family Resilience Assessment Scale, the Perceived Social Support Scale, and the Zarit Caregiver Burden Interview, while breast cancer survivors completed the shortened Chinese version of the Connor-Davidson Resilience Scale. Structural equation modeling was used to evaluate the relationships among social support, family resilience, survivors' individual resilience, and caregiver burden. Results Caregiver burden (45.76 ± 14.66) was found to be severe. Social support, family resilience, and individual resilience were significantly negatively associated with caregiver burden (β = − 0.421, P < 0.001; β = − 0.208, P < 0.001; and β = − 0.444, P < 0.001, respectively). Individual resilience not only partially mediated the relationship between family resilience and caregiver burden (b = − 0.052; 95% confidence interval, − 0.110, − 0.018), but also partially mediated the relationship between support and caregiver burden (b = − 0.045; 95% confidence interval, − 0.102, − 0.011). Conclusions The findings suggest that higher social support, family resilience, and individual resilience tend to ease caregivers' burden. Healthcare workers should have an in-depth understanding of the care needs of survivors, actively contact social security departments and social organizations to provide financial, technical, and emotional support, and provide family-based care-skills training and psychological counseling to reduce spousal caregivers' burdens.