Colorectal cancer (CRC) is one of the most common malignant tumors in the world. Tumor microenvironment (TME) plays a crucial role in the development of CRC. With the deep understanding of TME function, growing studies have demonstrated that alteration in tumor-infiltrating immune cells (TICs) and gene expressions are associated with clinical outcomes of various tumors. In this study, we aimed to recognize critical prognostic genes involved in immune states in TME of CRC. Hence, the proportion of TICs and the number of immune and stromal components in CRC samples from TCGA datasets were calculated by the use of CIBERSORT and ESTIMATE calculation methods. Different assays were applied to collect differential expression genes (DEGs) shared by the ImmuneScore and StromalScore. DEGs were further analyzed by the use of univariate Cox regression. Our attention focused on neuron navigator 3 (NAV3) which was highly expressed in CRC specimens and associated with an advanced clinical stage and poor prognosis of CRC patients. KEGG assays revealed that NAV3 may be involved in Alzheimer disease, amyotrophic lateral sclerosis, Huntington disease, FoxO signaling pathway, and human papillomavirus infection. Correlation assays showed that macrophage M0 and B cells memory, NK cells activated, dendritic cells resting, T cells CD4 memory activated, and T cells CD8 were correlated with NAV3 expression, indicating that NAV3 may represent the immune status of TME. Finally, RT-PCR confirmed that NAV3 expression was distinctly increased in CRC cells, and its knockdown suppressed the proliferation of CRC cells. Overall, NAV3 could be used as a novel predictor for TME of CRC and might be a novel prognostic biomarker. In the future, drugs targeting NAV3 might be developed as a potential immunotherapy for CRC patients.