Adaptive immune responses play an important role in the clinical course of SARS-CoV-2 infection. While evaluations of the virus-specific defense often focus on the humoral response, cellular immunity is crucial for the successful control of infection, with the early development of cytotoxic T cells being linked to efficient viral clearance. Vaccination against SARS-CoV-2 induces both CD4+ and CD8+ T cell responses and permits protection from severe COVID-19, including infection with the currently circulating variants of concern. Nevertheless, in immunocompromised individuals, first data imply significantly impaired SARS-CoV-2-specific immune responses after both natural infection and vaccination. Hence, these high-risk groups require particular consideration, not only in routine clinical practice, but also in the development of future vaccination strategies. In order to assist physicians in the guidance of immunocompromised patients, concerning the management of infection or the benefit of (booster) vaccinations, this review aims to provide a concise overview of the current knowledge about SARS-CoV-2-specific cellular immune responses in the vulnerable cohorts of cancer patients, people living with HIV (PLWH), and solid organ transplant recipients (SOT). Recent findings regarding the virus-specific cellular immunity in these differently immunocompromised populations might influence clinical decision-making in the future.