The plasticity of cancer cells allows them to switch between different migration modes, promoting their invasion into the extracellular matrix (ECM) and hence increasing the risks of metastasis. Epithelial-to-mesenchymal transitions (EMT) and unjamming transitions provide two distinct pathways for cancer cells to become invasive, but it is still unclear to what extent these pathways are connected. Here we addressed this question by performing 3D spheroid invasion assays of lung adenocarcinoma (A549, epithelial) and melanoma (MV3, mesenchymal-like) cancer cell lines in collagen-based hydrogels, where we varied both the invasive character of the cells (using Transforming Growth Factor (TGF)-$\beta$ to promote EMT and matrix metalloprotease (MMP) inhibition to block cell-mediated matrix degradation) and the porosity of the matrix. Using a quantitative image analysis method to track spheroid invasion, we discovered that the onset time of invasion mostly depended on the matrix porosity and corresponded with vimentin levels, while the subsequent spheroid expansion rate mostly depended on metalloprotease MMP1 levels and thus cell-matrix interaction. Morphological analysis revealed that spheroids displayed solid-like (non-invasive) behavior in small-pore hydrogels and switched to fluid-like (strand-based) or gas-like (disseminating cells) phases in large-pore hydrogels and when cells were more mesenchymal-like. Our findings are consistent with unjamming transitions as a function of cell motility and matrix confinement predicted in recent models for cancer invasion, but show that cell motility and matrix confinement are coupled via EMT-dependent matrix degradation.