Colorectal cancer (CRC) is a heterogeneous disease, with varying clinical presentations and patient prognosis. Different molecular subgroups of CRC should be treated differently and therefore, must be better characterized. Organoid culture has recently been suggested as a good model to reflect the heterogeneous nature of CRC. However, organoid cultures cannot be established from all CRC tumors. The study examines which CRC tumors are more likely to generate organoids and thus benefit from ex vivo organoid drug testing. Long-term organoid cultures from 22 out of 40 CRC tumor specimens were established. It was found that organoid cultures were more difficult to establish from tumors characterized as microsatellite instable (MSI), BRAF-mutated, poorly differentiated and/or of a mucinous type. This suggests that patients with such tumors are less likely to benefit from ex vivo organoid drug testing, but it may also suggest biological difference in tumor growth. RNA sequencing analysis of tumor sections revealed that the in vivo maintenance of these non-organoid-forming tumors depends on factors related to inflammation and pathogen exposure. Furthermore, using TCGA data we could show a trend towards a worse prognosis for patients with organoid-forming tumors, suggesting also clinical differences. Results suggest that organoids are more difficult to establish from tumors characterized as MSI, BRAF-mutated, poorly differentiated and/or of a mucinous type. We further suggest that the maintenance of cell growth of these tumors in vivo may be promoted by immune-related factors and other stromal components within the tumor microenvironment.