Cellular subpopulations within the colorectal tumor microenvironment (TME) include CD3 + and CD8 + lymphocytes, CD68 + and CD163 + macrophages, and tumor buds (TBs), all of which have known prognostic significance in stage II colorectal cancer. However, the prognostic relevance of their spatial interactions remains unknown. Here, by applying automated image analysis and machine learning approaches, we evaluate the prognostic significance of these cellular subpopulations and their spatial interactions. Resultant data, from a training cohort retrospectively collated from Edinburgh, UK hospitals (n = 113), were used to create a combinatorial prognostic model, which identified a subpopulation of patients who exhibit 100% survival over a 5-year follow-up period. The combinatorial model integrated lymphocytic infiltration, the number of lymphocytes within 50-μm proximity to TBs, and the CD68 + /CD163 + macrophage ratio. This finding was confirmed on an independent validation cohort, which included patients treated in Japan and Scotland (n = 117). This work shows that by analyzing multiple cellular subpopulations from the complex TME, it is possible to identify patients for whom surgical resection alone may be curative.