The extracellular matrix is a structure composed of many molecules, including fibrillar (types I, II, III, V, XI, XXIV, XXVII) and non-fibrillar collagens (mainly basement membrane collagens: types IV, VIII, X), non-collagenous glycoproteins (elastin, laminin, fibronectin, thrombospondin, tenascin, osteopontin, osteonectin, entactin, periostin) embedded in a gel of negatively charged water-retaining glycosaminoglycans (GAGs) such as non-sulfated hyaluronic acid (HA) and sulfated GAGs which are linked to a core protein to form proteoglycans (PGs). This highly dynamic molecular network provides critical biochemical and biomechanical cues that mediate the cell–cell and cell–matrix interactions, influence cell growth, migration and differentiation and serve as a reservoir of cytokines and growth factors’ action. The breakdown of normal ECM and its replacement with tumor ECM modulate the tumor microenvironment (TME) composition and is an essential part of tumorigenesis and metastasis, acting as key driver for malignant progression. Abnormal ECM also deregulate behavior of stromal cells as well as facilitating tumor-associated angiogenesis and inflammation. Thus, the tumor matrix modulates each of the classically defined hallmarks of cancer promoting the growth, survival and invasion of the cancer. Moreover, various ECM-derived components modulate the immune response affecting T cells, tumor-associated macrophages (TAM), dendritic cells and cancer-associated fibroblasts (CAF). This review article considers the role that extracellular matrix play in breast cancer. Determining the detailed connections between the ECM and cellular processes has helped to identify novel disease markers and therapeutic targets.