The tumor microenvironment (TME) is composed of tumor cells, blood/lymphatic vessels, the tumor stroma, and tumor-infiltrating myeloid precursors (TIMPs) as a sophisticated pathological system to provide the survival environment for tumor cells and facilitate tumor metastasis. In TME, TIMPs, mainly including tumor-associated macrophage (TAM), tumor-associated dendritic cells (DCs), and myeloid-derived suppressor cells (MDSCs), play important roles in repressing the antitumor activity of T cell or other immune cells. Therefore, targeting those cells would be one novel efficient method to retard cancer progression. Numerous studies have shown that traditional Chinese medicine (TCM) has made extensive research in tumor immunotherapy. In the review, we demonstrate that Chinese herbal medicine (CHM) and its components induce tumor cell apoptosis, directly inhibiting tumor growth and invasion. Further, we discuss that TCM regulates TME to promote effective antitumor immune response, downregulates the numbers and function of TAMs/MDSCs, and enhances the antigen presentation ability of mature DCs. We also review the therapeutic effects of TCM herbs and their ingredients on TIMPs in TME and systemically analyze the regulatory mechanisms of TCM on those cells to have a deeper understanding of TCM in tumor immunotherapy. Those investigations on TCM may provide novel ideas for cancer treatment.