Adipose tissue (AT) is comprised of a diverse number of cell types, including adipocytes, stromal cells, endothelial cells, and infiltrating leukocytes. Adipose stromal cells (ASCs) are a mixed population containing adipose progenitor cells (APCs) as well as fibro-inflammatory precursors and cells supporting the vasculature. There is growing evidence that the ability of ASCs to renew and undergo adipogenesis into new, healthy adipocytes is a hallmark of healthy fat, preventing disease-inducing adipocyte hypertrophy and the spillover of lipids into other organs, such as the liver and muscles. However, there is building evidence indicating that the ability for ASCs to self-renew is not infinite. With rates of ASC proliferation and adipogenesis tightly controlled by diet and the circadian clock, the capacity to maintain healthy AT via the generation of new, healthy adipocytes appears to be tightly regulated. Here, we review the contributions of ASCs to the maintenance of distinct adipocyte pools as well as pathogenic fibroblasts in cancer and fibrosis. We also discuss aging and diet-induced obesity as factors that might lead to ASC senescence, and the consequences for metabolic health.