Understanding interactions between tumor and the host immune system holds great promise to uncover biomarkers for targeted therapies and clinical outcomes. However, systematical analysis of immune signatures in esophageal squamous cell carcinoma (ESCC) remains largely unstudied. In this study, immune signatures containing 708 immune related genes were curated from mRNA microarray data with tumor and paired normal tissues from 119 ESCC patients. Differential expression and survival analysis were performed with validations from Human Protein Atlas and an independent cohort of 110 ESCC patients by immunohistochemistry staining. We identified a total of 186 significantly dysregulated genes in ESCC, including downregulated genes SPINK5, IL1RN and upregulated genes SPP1 and PLAU, which were further confirmed in Human Protein Atlas data. Moreover, nine immune related genes (ABL1, ATF2, ATG5, C6, CD38, HMGB1, ICOSLG, IL12RB2 and PLAU) were significantly associated with patients' overall survival, among which, prognostic model was built including three independent factors ABL1, CD38 and ICOSLG. Validation by immunohistochemistry staining suggested that combination with tumor infiltrated CD4+ and CD8+ T lymphocytes would yield higher performance in distinguishing cases as high or low risk of unfavorable prognosis. In summary, we profiled the immune status in ESCC and established predictive and prognostic factors for ESCC, which could reflect immune disorders within tumor microenvironments and independently distinguish patients with a high risk of reduced survival, providing novel predictive and therapeutic targets for ESCC patients in the future.