Checkpoint inhibitors are widely used immunotherapies for advanced cancer. Nonetheless, checkpoint inhibitors have a relatively low response rate, work in a limited range of cancers, and have some unignorable side effects. Checkpoint inhibitors aim to reinvigorate exhausted or suppressed T cells in the tumor microenvironment (TME). However, the TME contains various other immune cell subsets that interact to determine the fate of cytotoxic T cells. Activation of cytotoxic T cells is initiated by antigen cross-presentation of dendritic cells. Dendritic cells could also release chemokines and cytokines to recruit and foster T cells. B cells, another type of antigen-presenting cell, also foster T cells and can produce tumor-specific antibodies. Neutrophils, a granulocyte cell subset in the TME, impede the proliferation and activation of T cells. The TME also consists of cytotoxic innate natural killer cells, which kill tumor cells efficiently. Natural killer cells can eradicate major histocompatibility complex I-negative tumor cells, which escape cytotoxic T cell–mediated destruction. A thorough understanding of the immune mechanism of the TME, as reviewed here, will lead to further development of more powerful therapeutic strategies. We have also reviewed the clinical outcomes of patients treated with drugs targeting these immune cells to identify strategies for improvement and possible immunotherapy combinations.