There is evidence suggesting that immune genes play pivotal roles in the development and progression of colorectal cancer (CRC). Colorectal carcinoma patient data from The Cancer Genome Atlas (TCGA) and the Gene Expression Omnibus (GEO) were randomly classified into a training set, a test set, and an external validation set. Differentially expressed gene (DEG) analyses, univariate Cox regression, and the least absolute shrinkage and selection operator (LASSO) were used to identify survival-associated immune genes and develop a prognosis model. Receiver operating characteristic (ROC) analysis and principal component analysis (PCA) were used to evaluate the discrimination of the risk models. The model genes predicted were verified using the Human Protein Atlas (HPA) databases, colorectal cell lines, and fresh CRC and adjacent tissues. To understand the relationship between IRGs and immune invasion and the TME, we analyzed the content of immune cells and scored the TME using CIBERSORT and ESTIMATE algorithms. Finally, we predicted the potential sensitive chemotherapeutic drugs in different risk score groups by the Genomics of Drug Sensitivity in Cancer (GDSC). A total of 491 IRGs were screened, and 14 IRGs were identified to be significantly related to overall survival (OS) and applied to construct an immune-related gene (IRG) prognostic signature (IRGSig) for CRC patients. Calibration plots showed that nomograms have powerful predictive ability. PCA and ROC analysis further verified the predictive value of this fourteen-gene prognostic model in three independent databases. Furthermore, we discovered that the tumor microenvironment changed significantly during the tumor development process, from early to middle to late stage, which may be an essential factor for tumor deterioration. Finally, we selected six commonly used chemotherapeutic drugs that have the potential to be useful in the treatment of CRC. Altogether, immune genes were used to construct a prognosis model for CRC patients, and a variety of methods were used to test the accuracy of this model. In addition, we explored the immune mechanisms of CRC through immune cell infiltration and TME in CRC. Furthermore, we assessed the therapeutic sensitivity of many commonly used chemotherapeutic medicines in individuals with varying risk factors. Finally, the immune risk model and immune mechanism of CRC were thoroughly investigated in this paper.