T cells in colorectal cancer (CRC) are associated with improved survival. However, checkpoint immunotherapies antagonizing the suppression of these cells are ineffective in the great majority of patients. To better understand the immune cell regulation in CRC, we compared tumor-associated T lymphocytes and macrophages to the immune cell infiltrate of normal mucosa. Human colorectal tumor specimen and tumor-distant normal mucosa tissues of the same patients were collected. Phenotypes and functionality of tissue-derived T cells and macrophages were characterized using immunohistochemistry, RNA in situ hybridization, and multiparameter flow cytometry. CRC contained significantly higher numbers of potentially immunosuppressive CD39 and Helios-expressing regulatory T cells in comparison to normal mucosa. Surprisingly, we found a concomitant increase of pro-inflammatory IFNγ -producing T cells. PD-L1+ stromal cells were decreased in the tumor tissue. Macrophages in the tumor compared to tumor-distant normal tissue appear to have an altered phenotype, identified by HLA-DR, CD14, CX3CR1, and CD64, and tolerogenic CD206+ macrophages are quantitatively reduced. The prognostic effect of these observed differences between distant mucosa and tumor tissue on the overall survival was examined using gene expression data of 298 CRC patients. The combined gene expression of increased FOXP3, IFNγ, CD14, and decreased CD206 correlated with a poor prognosis in CRC patients. These data reveal that the CRC microenvironment promotes the coexistence of seemingly antagonistic suppressive and pro-inflammatory immune responses and might provide an explanation why a blockade of the PD1/PD-L1 axis is ineffective in CRC. This should be taken into account when designing novel treatment strategies.