Both immune profiling and tumor budding significantly correlate with colorectal cancer patient outcome but are traditionally reported independently. This study evaluated the association and interaction between lymphocytic infiltration and tumor budding, coregistered on a single slide, in order to determine a more precise prognostic algorithm for patients with stage II colorectal cancer. Multiplexed immunofluorescence and automated image analysis were used for the quantification of CD3+CD8+ T cells, and tumor buds (TBs), across whole slide images of three independent cohorts (training cohort: n = 114, validation cohort 1: n = 56, validation cohort 2: n = 62). Machine learning algorithms were used for feature selection and prognostic risk model development. High numbers of TBs [HR = 5.899; 95% confidence interval (CI) 1.875–18.55], low CD3+ T-cell density (HR = 9.964; 95% CI, 3.156–31.46), and low mean number of CD3+CD8+ T cells within 50 μm of TBs (HR = 8.907; 95% CI, 2.834–28.0) were associated with reduced disease-specific survival. A prognostic signature, derived from integrating TBs, lymphocyte infiltration, and their spatial relationship, reported a more significant cohort stratification (HR = 18.75; 95% CI, 6.46–54.43), than TBs, a lymphocytic infiltration score, or pT stage. This was confirmed in two independent validation cohorts (HR = 12.27; 95% CI, 3.524–42.73; HR = 15.61; 95% CI, 4.692–51.91). The investigation of the spatial relationship between lymphocytes and TBs within the tumor microenvironment improves accuracy of prognosis of patients with stage II colorectal cancer through an automated image analysis and machine learning workflow.