Esophageal adenocarcinoma (EAC) is an aggressive cancer with poor prognosis, and incidence is increasing rapidly in the Western world. Measurement of immune markers has been shown to have prognostic significance in a growing number of cancers, but whether this is true for EAC has yet to be evaluated. This study aimed to characterize HLA-DR expression in the esophagus across the inflammation to cancer progression sequence and to assess the prognostic significance of HLA-DR expression in EAC. Tissue microarrays (TMA) were constructed from esophageal tissue taken from patients at different stages in the cancer progression sequence; normal, esophagitis, Barrett’s esophagus (BE), low- and high-grade dysplasia (LGD, HGD) and EAC. HLA-DR expression in tissue epithelium and stroma was assessed by immunohistochemistry. HLA-DR expression increased early in the inflammation to cancer progression sequence; with higher expression detected in esophagitis and BE compared to normal tissue. Patients with low (<50%) HLA-DR expression in the EAC tumor epithelium had significantly worse survival outcomes, compared to those with high expression, in both the tumor core (hazard ratio, HR = 2.178, p = 0.024, n = 70) and leading edge (HR = 2.86, p = 0.013, n = 41). Multivariate analysis demonstrated that low HLA-DR expression in leading edge tumor epithelium was an independent predictor of poor survival, associated with a 2.8-fold increase in disease-associated death (p = 0.023). This study shows that HLA-DR is an independent prognostic marker in EAC tumor epithelium. This may have implications for patient stratification strategies as well as EAC tumor immunology.Electronic supplementary materialThe online version of this article (doi:10.1007/s00262-017-1983-1) contains supplementary material, which is available to authorized users.