Adipose tissue is a complex endocrine organ, with a role in obesity and cancer. Adipose tissue is generally linked to excessive body fat, and it is well known that the female breast is rich in adipose tissue. Hence, one can wonder: what is the role of adipose tissue in the breast and why is it required? Adipose tissue as an organ consists of adipocytes, an extracellular matrix (ECM) and immune cells, with a significant role in the dynamics of breast changes throughout the life span of a female breast from puberty, pregnancy, lactation and involution. In this review, we will discuss the importance of breast adipose tissue in breast development and its involvement in breast changes happening during pregnancy, lactation and involution. We will focus on understanding the biology of breast adipose tissue, with an overview on its involvement in the various steps of breast cancer development and progression. The interaction between the breast adipose tissue surrounding cancer cells and vice-versa modifies the tumor microenvironment in favor of cancer. Understanding this mutual interaction and the role of breast adipose tissue in the tumor microenvironment could potentially raise the possibility of overcoming breast adipose tissue mediated resistance to therapies and finding novel candidates to target breast cancer.