Mammary adipose tissue (AT) is necessary for breast epithelium. However, in breast cancer (BC), cell-cell interactions are deregulated as the tumor chronically modifies AT microenvironment. In turn, breast AT evolves to accommodate the tumor, and to participate to its dissemination. Among AT cells, adipocytes and their precursor mesenchymal stem cells (MSCs) play a major role in supporting tumor growth and dissemination. They provide energy supplies and release a plethora of factors involved in cancer aggressiveness. Here, we discuss the main molecular mechanisms underlining the interplay between adipose (adipocytes and MSCs) and BC cells. Following close interactions with BC cells, adipocytes lose lipids and change morphology and secretory patterns. MSCs also play a major role in cancer progression. While bone marrow MSCs are recruited by BC cells and participate in metastatic process, mammary AT-MSCs exert a local action by increasing the release of cytokines, growth factors and extracellular matrix components and become principal actors in cancer progression. Common systemic metabolic diseases, including obesity and diabetes, further modify the interplay between AT and BC. Indeed, metabolic perturbations are accompanied by well-known alterations of AT functions, which might contribute to worsen cancer phenotype. Here, we highlight how metabolic alterations locally affect mammary AT and interfere with the molecular mechanisms of bidirectional communication between adipose and cancer cells.