Backgrounds: The relationship between sarcopenia, characterized by loss of muscle mass and strength, and survival outcomes of esophageal cancer is controversial. This study aimed to assess the effect of sarcopenia and skeletal muscle loss on overall survival (OS) and recurrence-free survival (RFS) of esophageal cancer patients. Methods: We retrospectively collected the medical records of 248 male patients diagnosed with squamous cell esophageal cancer and who underwent neoadjuvant chemoradiotherapy (NACRT) followed by surgery. We measured the cross-sectional area of the skeletal muscle at the L3 vertebra level using computed tomography images and calculated the skeletal muscle index (SMI). Sarcopenia was defined as SMI <52.4 cm2/m2, and excessive muscle loss was defined as SMI change <−10.0%/50 days during NACRT. Moreover, laboratory test results, such as albumin, prognostic nutritional index (PNI), neutrophil-to-lymphocyte ratio (NLR), and platelet-to-lymphocyte ratio (PLR) before and after NACRT, were collected. Results: In the univariable Cox analysis, pre- (p = 0.689) and post-radiotherapy (RT) sarcopenia (p = 0.669) were not associated with OS. However, excessive muscle loss had a significant association with OS in both the univariable and multivariable analyses (all p = 0.001). Excessive muscle loss was also related to RFS in both the univariable (p = 0.011) and multivariable (p = 0.022) Cox analysis. Patients with excessive muscle loss had significantly lower levels of post-RT albumin (p < 0.001) and PNI (p < 0.001), higher levels of post-RT NLR (p = 0.031) and PLR (p = 0.071), larger decrease in albumin (p < 0.001) and PNI (p < 0.001) after NACRT, and larger increase in NLR (p = 0.051) and PLR (p = 0.088) after NACRT than in those with non-excessive muscle loss. Conclusion: Excessive muscle loss rather than pre- and post-RT sarcopenia was a significant prognostic factor for OS and RFS, and it was also related to nutritional and inflammatory markers.