Although most cancer types have been viewed as immunologically silent until recently, it has become increasingly clear that the immune system plays key roles in the course of tumor development. Remarkable progress towards understanding cancer immunogenicity and tumor-immune system interactions has revealed important implications for the design of novel immune-based therapies. Natural immune responses, but also therapeutic interventions, can modulate the tumor phenotype due to selective outgrowth of resistant subtypes. This is the result of heterogeneity of tumors, with genetic instability as a driving force, and obviously changes the immunogenicity of tumors. In this review, we discuss the immunogenicity of colorectal cancer (CRC) in relation to tumor development and treatment. As most tumors, CRC activates the immune system in various ways, and is also capable of escaping recognition and elimination by the immune system. Tumor-immune system interactions underlie the balance between immune control and immune escape, and may differ in primary tumors, in the circulation, and in liver metastases of CRC. Since CRC immunogenicity varies between tumors and individuals, novel immune-based therapeutic strategies should not only anticipate the molecular profile, but also the immunological profile of a specific tumor.