Background Fibroids and endometriosis are sex hormone-mediated and exhibit cancer-like behavior. Breast cancer may be more common in women who have had these conditions, but the literature is conflicting and does not always address factors like hysterectomy/oophorectomy status, race/ethnicity, menopause, and hormone receptor subtypes. Methods Data are from the Sister Study, a cohort of 50,884 U.S. women enrolled in 2003-2009 and followed through 2020. Cox proportional hazards models with time-varying exposures and covariates assessed the relationship of fibroids or endometriosis with breast cancer. Logistic regression examined the association with estrogen receptor (ER) status among cases. Results Fibroids (19,932 cases) were positively associated with breast cancer (fully adjusted hazard ratio [HR]: 1.07, 95% confidence interval [CI]: 1.01-1.14), notably among Black participants (HR 1.34, 95% CI: 1.07-1.69) and women who had a hysterectomy (HR: 1.18, 95% CI: 1.05-1.31). Endometriosis (3,970 cases) was not associated with breast cancer (HR: 0.99, 95% CI: 0.91-1.08). Among 4,419 breast cancer cases, fibroids were positively associated with ER+ subtypes (odds ratio [OR]: 1.34, 95% CI: 1.10-1.65), while endometriosis was negatively associated with ER+ subtypes (OR: 0.78; 95% CI: 0.61-1.01). Conclusions We observed a modest positive association between fibroids and breast cancer, particularly ER+ breast cancer. No relationship with endometriosis and breast cancer incidence was found. Impact Fibroids, even in those with a family history of breast cancer, might modify breast cancer risk stratification tools. Future studies should further assess this link and interrogate shared risk factors.