Background Esophageal cancer (ESCA) is one of the deadliest solid malignancies with worse survival in the world. The poor prognosis of ESCA is not only related to malignant cells, but also affected by the microenvironment. We aimed to establish prognostic signature consisting of immune genes to predict the survival outcome of patients and estimate the prognosis value of infiltrating immune cells in tumor microenvironment (TME). Methods Based on integrated analysis of gene expression profiling and immune gene database, differentially immune-related genes were filtered out. Then, stepwise Cox regression analysis was applied to identify survival related immune genes and construct prognosis signature. Functional enrichment analysis was performed to explore biology function. Kaplan-Meier (K-M) curves and receiver operating characteristic (ROC) curves were performed to validate the predictive effect of predictive signature. We also verified the clinical value of prognostic signature under the influence of different clinical parameters. For deeper analysis, we evaluated the correlation between prognosis signature and infiltrating immune cells by Tumor Immune Estimation Resource (TIMER) and CIBERSORT. Results Finally, we identified 303 differentially immune genes as candidate and constructed immune prognosis signature composed of six immune genes. Furthermore, we observed that the prognosis signature was enriched in cytokine-mediated signaling pathway, lymphocyte activation, immune effector process, cancer pathway, NF-kappa B signaling pathway. K-M survival curves showed that the prognosis signature indeed have good predictive ability in entire ESCA set ( P =0.003), validation set 1 ( P =0.008) and validation set 2 ( P =0.036). The area under the curve (AUC) of ROC curves validated the predictive accuracy of immune signature in three cohorts (AUC=0.757, 0.800 and 0.701), respectively. In addition, we identified the prognosis value of infiltrating-immune cells including activated memory CD4 T cells, T cells follicular helper cells and monocytes and provided a landscape of TME. Conclusions The results indicated that immune prognosis signature can be a novel biomarker to predict survival outcome, which can provide new targets for immunotherapy and individualized therapies in ESCA and open up a new prospect for improving the prognosis of ESCA patients in the era of immunotherapy.