Large primary tumor volume has been identified as a poor prognostic factor of esophageal squamous cell carcinoma (ESCC) treated with definitive concurrent chemoradiotherapy (CCRT). However, when neoadjuvant CCRT and surgery are adopted, the prognostic impact of primary tumor and lymph node (LN) volume on clinical outcomes in ESCC remains to be elucidated. This study included 107 patients who received neoadjuvant CCRT and surgery for ESCC. The volume of the primary tumor and LN was measured using radiotherapy planning computed tomography scans, and was correlated with overall survival (OS), disease-free survival (DFS), and cancer failure pattern. The median OS was 24.2 months (IQR, 11.1–93.9) after a median follow-up of 18.4 months (IQR, 8.1–40.7). The patients with a baseline LN volume > 7.7 ml had a significantly worse median OS compared to those with smaller LN volume (18.8 vs. 46.9 months, p = 0.049), as did those with tumor regression grade (TRG) 3–5 after CCRT (13.9 vs. 86.7 months, p < 0.001). However, there was no association between OS and esophageal tumor volume (p = 0.363). Multivariate analysis indicated that large LN volume (HR 1.753, 95% CI 1.015–3.029, p = 0.044) and high TRG (HR 3.276, 95% CI 1.556–6.898, p = 0.002) were negative prognostic factors for OS. Furthermore, large LN volume was linked to increased locoregional failure (p = 0.033) and decreased DFS (p = 0.041). In conclusion, this study demonstrated that large LN volume is correlated with poor OS, DFS, and locoregional control in ESCC treated with neoadjuvant CCRT and esophagectomy.