Ubiquitin-specific protease 4 (USP4) represents a potential oncogene involved in various human cancers. Nevertheless, the biological roles and precise mechanism of USP4 in esophageal squamous cell carcinoma (ESCC) progression are not understood. Here, USP4 expression was found to be markedly upregulated in ESCC tumor tissues and cells. Loss- and gain-of-function assays suggested that USP4 silencing inhibited ESCC cell proliferation, migration, and invasion, while USP4 overexpression promoted these behaviors. Consistently, USP4 silencing repressed tumor growth and metastasis in an ESCC nude mouse model in vivo. As a target molecule of USP4, transforming growth factor-β-activated kinase 1 (TAK1) also showed high expression in ESCC. Moreover, we observed that USP4 specifically interacted with TAK1 and stabilized TAK1 protein levels via deubiquitination in ESCC cells. Importantly, USP4 promotes ESCC proliferation, migration, and invasion via the MEK/ERK signaling pathway and can be inhibited by U0126. Neutral red (NR), an inhibitor of USP4 can suppress ESCC progression in vitro and in vivo. Overall, this study revealed that USP4/TAK1 plays crucial roles in ESCC progression by modulating proliferation, migration, and invasion, and USP4 might be a potential therapeutic target in ESCC.