The tumor microenvironment (TME) is a complex, dynamic battlefield for both immune cells and tumor cells. The advent of the immune checkpoint inhibitors (ICI) since 2011, such as the anti-cytotoxic T-lymphocyte associated protein (CTLA)-4 and anti-programmed cell death receptor (PD)-(L)1 antibodies, provided powerful weapons in the arsenal of cancer treatments, demonstrating unprecedented durable responses for patients with many types of advanced cancers. However, the response rate is generally low across tumor types and a substantial number of patients develop acquired resistance. These primary or acquired resistance are attributed to various immunosuppressive elements (soluble and cellular factors) and alternative immune checkpoints in the TME. Therefore, a better understanding of the TME is absolutely essential to develop therapeutic strategies to overcome resistance. Numerous clinical studies are underway using ICIs and additional agents that are tailored to the characteristics of the tumor or the TME. Some of the combination treatments are already approved by the Food and Drug Administration (FDA), such as platinum-doublet chemotherapy, tyrosine kinase inhibitor (TKI) -targeting vascular endothelial growth factor (VEGF) combined with anti-PD-(L)1 antibodies or immuno-immuno combinations (anti-CTLA-4 and anti-PD-1). In this review, we will discuss the key immunosuppressive cells, metabolites, cytokines or chemokines, and hypoxic conditions in the TME that contribute to tumor immune escape and the prospect of relevant clinical trials by targeting these elements in combination with ICIs.