Background: Breast cancer is the most common cancer and the second leading cause of cancer-related death amongst American women. Endocrine-disrupting chemicals (EDCs), especially bisphenol A (BPA) and phthalates, have adverse effects on human health. However, the association of BPA and phthalates with breast cancer remains conflicting. This study aims to investigate the association of BPA and phthalates with breast cancer. Methods: Correlative studies were identified by systematically searching three electronic databases, namely, PubMed, Web of Sciences, and Embase, up to November 2020. All data were analyzed using Stata 15.0. Results: A total of nine studies, consisting of 7820 breast cancer cases and controls, were included. The urinary phthalate metabolite mono-benzyl phthalate (MBzP) and mono-2-isobutyl phthalate (MiBP) were negatively associated with breast cancer (OR = 0.73, 95% CI: 0.60–0.90; OR = 0.75, 95% CI: 0.58–0.98, respectively). However, the overall ORs for BPA, mono-ethyl phthalate (MEP), mono-(2-ethyl-5-hydroxyhexyl) phthalate (MEHHP), mono-2-ethylhexyl phthalate (MEHP), mono-(2-ethyl-5-oxohexyl) phthalate (MEOHP), mono-(3-carboxypropyl) phthalate (MCPP), and mono-butyl phthalate (MBP) were 0.85 (95% CI: 0.69–1.05), 0.96 (95% CI: 0.62–1.48), 1.12 (95% CI: 0.88–1.42), 1.13 (95% CI: 0.74–1.73), 1.01 (95% CI: 0.74–1.40), 0.74 (95% CI: 0.48–1.14), and 0.80 (95% CI: 0.55–1.15), respectively, suggesting no significant association. The sensitivity analysis indicated that the results were relatively stable. Conclusion: Phthalate metabolites MBzP and MiBP were passively associated with breast cancer, whereas no associations were found between BPA, MEP, MEHHP, MEHP, MEOHP, MCPP, and MBP and breast cancer. More high-quality case-control studies or persuasive cohort studies are urgently needed to draw the best conclusions.