The extracellular matrix (ECM) plays a crucial role in regulating organ homeostasis. It provides mechanical and biochemical cues directing cellular behaviour and, therefore, has control over the progression of diseases such as cancer. Recent efforts have greatly enhanced our knowledge of the protein composition of the ECM and its regulators, the so-called matrisome, in healthy and cancerous tissues; yet, an overview of the common signatures and organ-specific ECM in cancer is missing. Here, we address this by taking a detailed approach to review why cancer grows in certain organs, and focus on the influence of the matrisome at primary and metastatic tumour sites. Our in-depth and comprehensive review of the current literature and general understanding identifies important commonalities and distinctions, providing insight into the biology of metastasis, which could pave the way to improve future diagnostics and therapies.