Background: We conducted a meta-analysis to quantitatively assess the association between asbestos exposure and esophageal cancer. Methods: We systematically collected articles from three electronic databases and calculated the pooled standardized mortality rate (SMR) from the meta-analysis. Subgroup analysis according to the type of asbestos exposure, follow-up years, sample size, industry classification, sex, and high-dose exposure was conducted. Results: From 242 studies, 34 cohort studies were included in our meta-analysis. Pooled SMR was positively associated with asbestos exposure and esophageal cancer (pooled SMR = 1.28; 95% confidence interval (CI) 1.19–1.38, p < 0.00001). In the subgroup analysis, (1) chrysolite, (2) four groups with follow-up over ten years, (3) the textile industry and shipyard, (4) both male and female, and (5) eight studies on highest asbestos exposure, all the subgroups showed significantly increased pooled SMRs. Conclusion: Asbestos exposure was significantly and positively associated with esophageal cancer, especially chrysolite. Considering the long latency period, we suggest that patients should be followed up for cancer, including esophageal cancer, for over ten years.