Immunotherapy has achieved efficacy for advanced colorectal cancer (CRC) patients with a mismatch-repair-deficient (dMMR) subtype. However, little immunotherapy efficacy was observed in patients with the mismatch repair-proficient (pMMR) subtype, and hence, identifying new immune therapeutic targets is imperative for those patients. In this study, transcriptome data of stage III/IV CRC patients were retrieved from the Gene Expression Omnibus database. The CIBERSORT algorithm was used to quantify immune cellular compositions, and the results revealed that M2 macrophage fractions were higher in pMMR patients as compared with those with the dMMR subtype; moreover, pMMR patients with higher M2 macrophage fractions experienced shorter overall survival (OS). Subsequently, weighted gene co-expression network analysis and protein–protein interaction network analysis identified six hub genes related to M2 macrophage infiltrations in pMMR CRC patients: CALD1, COL6A1, COL1A2, TIMP3, DCN, and SPARC. Univariate and multivariate Cox regression analyses then determined CALD1 as the independent prognostic biomarker for OS. CALD1 was upregulated specifically the in CMS4 CRC subtype, and single-sample Gene Set Enrichment Analysis (ssGSEA) revealed that CALD1 was significantly correlated with angiogenesis and TGF-β signaling gene sets enrichment scores in stage III/IV pMMR CRC samples. The Estimation of STromal and Immune cells in MAlignant Tumor tissues using Expression data (ESTIMATE) algorithm and correlation analysis revealed that CALD1 was significantly associated with multiple immune and stromal components in a tumor microenvironment. In addition, GSEA demonstrated that high expression of CALD1 was significantly correlated with antigen processing and presentation, chemokine signaling, leukocyte transendothelial migration, vascular smooth muscle contraction, cytokine–cytokine receptor interaction, cell adhesion molecules, focal adhesion, MAPK, and TGF-beta signaling pathways. Furthermore, the proliferation, invasion, and migration abilities of cancer cells were suppressed after reducing CALD1 expression in CRC cell lines. Taken together, multiple bioinformatics analyses and cell-level assays demonstrated that CALD1 could serve as a prognostic biomarker and a prospective therapeutic target for stage III/IV pMMR CRCs.