The majority of colorectal cancer (CRC) patients carry mutations in the APC gene, which lead to the unregulated activation of the Wnt pathway. Extracellular vesicles (EV) are considered potential therapeutic tools. Although CRC is a genetically heterogeneous disease, the significance of the intra-tumor heterogeneity in EV uptake of CRC cells is not yet known. By using mouse and patient-derived organoids, the currently available best model of capturing cellular heterogeneity, we found that Apc mutation induced the expression of interferon-induced transmembrane protein 1 (Ifitm1), a membrane protein that plays a major role in cellular antiviral responses. Importantly, organoids derived from IFITM1high CRC cells contained more proliferating cells and they had a markedly reduced uptake of fibroblast EVs as compared to IFITM1low/− cells. In contrast, there was no difference in the intensity of EV release between CRC subpopulations with high and low IFITM1 levels. Importantly, the difference in cell proliferation between these two subpopulations disappeared in the presence of fibroblast-derived EVs, proving the functional relevance of the enhanced EV uptake by IFITM1low CRC cells. Furthermore, inactivating IFITM1 resulted in an enhanced EV uptake, highlighting the importance of this molecule in establishing the cellular difference for EV effects. Collectively, we identified CRC cells with functional difference in their EV uptake ability that must be taken into consideration when using EVs as therapeutic tools for targeting cancer cells.