Cancer immunotherapy, largely represented by immune checkpoint inhibitors (ICIs), has led to substantial changes in preclinical cancer research and clinical oncology practice over the past decade. However, the efficacy and toxicity profiles of ICIs remain highly variable among patients, with only a fraction achieving a significant benefit. New combination therapeutic strategies are being investigated, and the search for novel predictive biomarkers is ongoing, mainly focusing on tumor- and host-intrinsic components. Less attention has been directed to all the external, potentially modifiable factors that compose the exposome, including diet and lifestyle, infections, vaccinations, and concomitant medications, which could affect the immune system response and its activity against cancer cells. We hereby provide a review of the available clinical evidence elucidating the impact of host-extrinsic factors on ICI response and toxicity.