Average age and obesity prevalence are increasing globally. Both aging and obesity are characterized by profound systemic metabolic and immunologic changes and are cancer risk factors. The mechanisms linking age and body weight to cancer are incompletely understood, but recent studies have provided evidence that the anti-tumor immune response is reduced in both conditions, while responsiveness to immune checkpoint blockade, a form of cancer immunotherapy, is paradoxically intact. Dietary restriction, which promotes health and lifespan, may enhance cancer immunity. These findings illustrate that the systemic context can impact anti-tumor immunity and immunotherapy responsiveness. Here, we review the current knowledge of how age and systemic metabolic state affect the anti-tumor immune response, with an emphasis on CD8+ T cells, which are key players in anti-tumor immunity. A better understanding of the underlying mechanisms may lead to novel therapies enhancing anti-tumor immunity in the context of aging or metabolic dysfunction.