Tumor-associated immune cells have been shown to predict patient outcome in colorectal (CRC) and other cancers. Spatial digital image analysis-based cell quantification increases the informative power delivered by tumor microenvironment features and leads to new prognostic scoring systems. In this study we evaluated the intratumoral density of immunohistochemically stained CD8, CD20 and CD68 cells in 87 cases of CRC (48 were microsatellite stable, MSS, and 39 had microsatellite instability, MSI) in both the intratumoral tumor tissue and within the tumor-stroma interface zone (IZ) which was extracted by a previously developed unbiased hexagonal grid analytics method. Indicators of immune-cell gradients across the extracted IZ were computed and explored along with absolute cell densities, clinicopathological and molecular data, including gene mutation (BRAF, KRAS, PIK3CA) and MSI status. Multiple regression modeling identified (p < 0.0001) three independent prognostic factors: CD8+ and CD20+ Immunogradient indicators, that reflect cell migration towards the tumor, were associated with improved patient survival, while the infiltrative tumor growth pattern was linked to worse patient outcome. These features were combined into CD8-CD20 Immunogradient and immuno-interface scores which outperformed both tumor-node-metastasis (TNM) staging and molecular characteristics, and importantly, revealed high prognostic value both in MSS and MSI CRCs.