Cancer-related immunity has been identified as playing a key role in the outcome of colorectal cancer (CRC); however, the exact mechanisms are only partially understood. In this study, we evaluated a total of 242 surgical specimen of CRC patients using tissue microarrays and immunohistochemistry to evaluate tumor infiltrating immune cells (CD3, CD4, CD8, CD20, CD23, CD45 and CD56) and immune checkpoint markers (CTLA-4, PD-L1, PD-1) in systematically selected tumor regions and their corresponding lymph nodes, as well as in liver metastases. Additionally, an immune panel gene expression assay was performed on 12 primary tumors and 12 consecutive liver metastases. A higher number of natural killer cells and more mature B cells along with PD-1+ expressing cells were observed in the main tumor area as compared to metastases. A higher number of metastatic lymph nodes were associated with significantly lower B cell counts. With more advanced lymph node metastatic status, higher leukocyte—particularly T cell numbers—were observed. Eleven differentially expressed immune-related genes were found between primary tumors and liver metastases. Also, alterations of the innate immune response and the tumor necrosis factor superfamily pathways had been identified.