Breast cancer survivors treated with anti-estrogen therapies report weight gain and have an elevated risk of type 2 diabetes. Here, we show that current tamoxifen use associated with larger breast adipocyte diameter only in women with a BMI >30 kg/m2. To understand the mechanisms behind these clinical findings, we investigated the impact of estrogen deprivation and tamoxifen in a relevant pre-clinical model of obesity. Specifically, mature female mice were housed at thermoneutrality and fed either a low-fat/low-sucrose (LFLS) or a high-fat/high-sucrose (HFHS) diet. Consistent with the high expression of Esr1 observed in single-cell RNA sequencing of mesenchymal stem cells from adipose tissue, endocrine therapies induced adipose accumulation and preadipocyte expansion, but resulted in adipocyte progenitor depletion only in the context of HFHS. Consequently, 7-week endocrine therapy supported adipocyte hypertrophy and was associated with hepatic steatosis, hyperinsulinemia, insulin resistance, and glucose intolerance, particularly in HFHS fed females. Metformin or pioglitazone, glucose lowering drugs used to treat diabetes, prevented the effects of tamoxifen but not estrogen deprivation on adipocyte size and insulin resistance in HFHS-fed mice. This translational study suggests that endocrine therapies act via ER-alpha; to directly disrupt adipocyte progenitors and support adipocyte hypertrophy, leading to ectopic lipid deposition that may promote hyperinsulinemia, insulin resistance and type 2 diabetes. Interventions that target insulin action should be considered for some women receiving life-saving endocrine therapies for breast cancer.