Individual environmental contaminants have been associated with breast cancer; however, evaluations of multiple exposures simultaneously are limited. Herein, we evaluated associations between breast cancer summary stages and the Environmental Quality Index (EQI), which includes a range of environmental factors across five domains. The EQI (2000–2005) was linked to county-level age-standardized incidence rates (SIRs) obtained from the North Carolina Central Cancer Registry (2010–2014). Incidence rates and SIRs of total, in situ, localized, regional, and distant breast cancers were evaluated stratified by rural–urban status. In counties with poor environmental quality compared to those with good environmental quality, total breast cancer incidence was higher by 10.82 cases per 100,000 persons (95% CI 2.04, 19.60, p = 0.02). This association was most pronounced for localized breast cancer (β = 5.59, 95% CI 0.59, 10.58, p = 0.03). Higher incidence of early-stage disease (carcinoma in situ β = 5.25, 95% CI 2.34, 8.16, p = 0.00 and localized breast cancer β = 6.98, 95% CI 2.24, 11.73, p = 0.00) and total breast cancer (β = 11.44, 95% CI 3.01, 19.87, p = 0.01) occurred in counties with poor land quality, especially urban counties. Our analyses indicate significant associations between environmental quality and breast cancer incidence, which differ by breast cancer stage and urbanicity, identifying a critical need to assess cumulative environmental exposures in the context of cancer stage.