Population studies show worrisome trends towards earlier breast development, difficulty in breastfeeding, and increasing rates of breast cancer in young women. Multiple epidemiological studies have linked these outcomes with chemical exposures, and experimental studies have shown that many of these chemicals generate similar effects in rodents, often by disrupting hormonal regulation. These endocrine-disrupting chemicals (EDCs) can alter the progression of mammary gland (MG) development, impair the ability to nourish offspring via lactation, increase mammary tissue density, and increase the propensity to develop cancer. However, current toxicological approaches to measuring the effects of chemical exposures on the MG are often inadequate to detect these effects, impairing our ability to identify exposures harmful to the breast and limiting opportunities for prevention. This paper describes key adverse outcomes for the MG, including impaired lactation, altered pubertal development, altered morphology (such as increased mammographic density), and cancer. It also summarizes evidence from humans and rodent models for exposures associated with these effects. We also review current toxicological practices for evaluating MG effects, highlight limitations of current methods, summarize debates related to how effects are interpreted in risk assessment, and make recommendations to strengthen assessment approaches. Increasing the rigor of MG assessment would improve our ability to identify chemicals of concern, regulate those chemicals based on their effects, and prevent exposures and associated adverse health effects.