A total of 94 patients with colorectal cancer (CRC) were included in this study. Lymphocytic infiltration of CD45+ cells in the normal colon was more pronounced than that in the paired tumor stroma (p = 0.0008). The mean immunoscore of CD45+TILs was decreased in CRC compared with the controls (p = 0.0010). The percentage of CD3+ cells was higher in stage II than in stage IV (p = 0.0218) and showed a negative correlation with the TNM classification (r = -0.2867, p = 0.0109). The number of stromal CD4+TILs was higher in stage I than in stage III (p = 0.0116) and IV (p = 0.0104), and there was a negative correlation between this number and the stage (r = -0.3708, p = 0.0008). There was a positive correlation between the Ki-67 and CD45+ (r = 0.2468, p = 0.0294), CD3+ (r = 0.3822, p = 0.0006), and CD4+ cells (r = 0.5465, p < 0.0001). The levels of cancer-associated fibroblast (CAF) markers such as α-SMA, thrombin and fibronectin were significantly higher in CRC than in normal colonic mucosa. The immunohistochemical expression of α-SMA was negatively correlated with TILs, while fibronectin showed positive coexpression. A higher number of cells expressing IL-2Rα, PD-L1, CD33 and CD14 were found in colorectal adenocarcinomas than in controls. The number of CD14+ cells was also dependent on the TNM stage (p = 0.0444) and tumor budding (p = 0.0324). These findings suggest a suppressive impact of CRC on the adaptive immune response and emphasize the importance of CAFs in regulating tumor immunity.