Tumor microenvironment, including extracellular matrix (ECM) and stromal cells, is a key player during tumor development, from initiation, growth and progression to metastasis. During all of these steps, remodeling of matrix components occurs, changing its biochemical and physical properties. The global and basic cancer ECM model is that tumors are surrounded by activated stromal cells, that remodel physiological ECM to evolve into a stiffer and more crosslinked ECM than in normal conditions, thereby increasing invasive capacities of cancer cells. In this review, we show that this too simple model does not consider the complexity, specificity and heterogeneity of each organ and tumor. First, we describe the general ECM in context of cancer. Then, we go through five invasive and most frequent cancers from different origins (breast, liver, pancreas, colon, and skin), and show that each cancer has its own specific matrix, with different stromal cells, ECM components, biochemical properties and activated signaling pathways. Furthermore, in these five cancers, we describe the dual role of tumor ECM: as a protective barrier against tumor cell proliferation and invasion, and as a major player in tumor progression. Indeed, crosstalk between tumor and stromal cells induce changes in matrix organization by remodeling ECM through invadosome formation in order to degrade it, promoting tumor progression and cell invasion. To sum up, in this review, we highlight the specificities of matrix composition in five cancers and the necessity not to consider the ECM as one general and simple entity, but one complex, dynamic and specific entity for each cancer type and subtype.