Gut microbiota can have opposing functions from pro-tumorigenic to anti-tumorigenic effects. Increasing preclinical and clinical evidence suggests that the intestinal microbiota affects cancer patients’ response to immune checkpoint inhibitors (ICIs) immunotherapy, such as anti-programmed cell death protein 1 (PD-1) and its ligand (PD-L1) and anti-cytotoxic T lymphocyte-associated protein 4 (CTLA-4). Microbiota-induced inflammation possibly contributes to tumor growth and cancer development. Microbiota-derived metabolites can also be converted to carcinogenic agents related to genetic mutations and DNA damage in organs such as the colon. However, other attributes of microbiota, such as greater diversity and specific bacterial species and their metabolites, are linked to better clinical outcomes and potentially improved anti-tumor immunity. In addition, the intratumoral microbial composition strongly affects T-cell-mediated cytotoxicity and anti-tumor immune surveillance, adding more complexity to the cancer-microbiome-immune axis. Despite the emerging clinical evidence for the activity of the gut microbiota in immuno-oncology, the fundamental mechanisms of such activity are not well understood. This review provides an overview of underlying mechanisms by which the gut microbiota and its metabolites enhance or suppress anti-tumor immune responses. Understanding such mechanisms allows for better design of microbiome-specific treatment strategies to improve the clinical outcome in cancer patients undergoing systemic therapy.