Cancer incidence increases drastically with age. Of the many possible reasons for this, there is the accumulation of senescent cells in tissues and the loss of function and proliferation potential of immune cells, often referred to as immuno-senescence. Immune checkpoint inhibitors (ICI), by invigorating immune cells, have the potential to be a game-changers in the treatment of cancer. Yet, the variability in the efficacy of ICI across patients and cancer types suggests that several factors influence the success of such inhibitors. There is currently a lack of clinical studies measuring the impact of aging and senescence on ICI-based therapies. Here, we review how cellular senescence and aging, either by directly altering the immune system fitness or indirectly through the modification of the tumor environment, may influence the cancer-immune response.