Esophageal squamous cell carcinoma (ESCC) is one of the most common cancers in the world, with a high rate of morbidity. The invasion and metastasis of ESCC is the main reason for high mortality. More and more evidence suggests that metastasized cancer cells require cellular elements that contribute to ESCC tumor microenvironment (TME) formation. TME contains many immune cells and stromal components, which are critical to epithelial–mesenchymal transition, immune escape, angiogenesis/lymphangiogenesis, metastasis niche formation, and invasion/metastasis. In this review, we will focus on the mechanism of different microenvironment cellular elements in ESCC invasion and metastasis and discuss recent therapeutic attempts to restore the tumor-suppressing function of cells within the TME. It will represent the whole picture of TME in the metastasis and invasion process of ESCC.