Estrogen plays a role in the processes of tumorigenesis, metastasis, and drug resistance in estrogen receptor (ER)-positive breast cancer (BC). Whether estrogen contributes to ER-negative BC is unclear. Here, we aimed to investigate whether estrogen could stimulate the secretion of stromal-derived factor-1 (SDF-1α) by cancer-associated fibroblasts (CAFs) to promote the progression of ER-negative BC. We transplanted ER-negative BC cells into ovariectomized mice, which was followed by continuous injection of estrogen, and found that estrogen promoted the tumorigenesis of BC. Furthermore, High levels of SDF-1α and tumor-infiltrating myeloid-derived suppressor cells (MDSCs) were detected in the estrogen treatment group. Estrogen stimulates secretion of SDF-1α by CAFs extracted from BC patients. Recombinant SDF-1α could recruit MDSCs isolated from bone marrow cells of mice. In addition, the co-culture of CAFs and MDSCs demonstrated that the recruitment of MDSCs was increased when CAFs were exposed to estrogen. Using AMD3100 to block the SDF-1α/CXCR4 axis or gemcitabine to delete MDSCs, we observed that both of these agents could neutralize the effect of estrogen on tumorigenesis. Together, these results suggest that estrogen may promote the progression of ER-negative BC by stimulating CAFs to secrete SDF-1α, which can recruit MDSCs to the tumor microenvironment to exert tumor-promoting effects.