Tumor mutational burden (TMB) is considered a potential biomarker for predicting the response and effect of immune checkpoint inhibitors (ICIs). However, there are still inconsistent standards of gene panels using next-generation sequencing and poor correlation between the TMB genes, immune cell infiltrating, and prognosis. We applied text-mining technology to construct specific TMB-associated gene panels cross various cancer types. As a case exploration, Pearson’s correlation between TMB genes and immune cell infiltrating was further analyzed in colorectal cancer. We then performed LASSO Cox regression to construct a prognosis predictive model and calculated the risk score of each sample for receiver operating characteristic (ROC) analysis. The results showed that the assessment of TMB gene panels performed well with fewer than 500 genes, highly mutated genes, and the inclusion of synonymous mutations and immune regulatory and drug-target genes. Moreover, the analysis of TMB differentially expressed genes (DEGs) suggested that JAKMIP1 was strongly correlated with the gene expression level of CD8+ T cell markers in colorectal cancer. Additionally, the prognosis predictive model based on 19 TMB DEGs reached AUCs of 0.836, 0.818, and 0.787 in 1-, 3-, and 5-year OS models, respectively (C-index: 0.810). In summary, the gene panel performed well and TMB DEGs showed great potential value in immune cell infiltration and in predicting survival.