Coronaviruses regularly cause outbreaks of zoonotic diseases characterized by severe pneumonia. The new coronavirus, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has caused the global pandemic disease COVID-19 that began at the end of 2019 and spread rapidly owing to its infectious nature and rapidly progressing pneumonia. Although the infectivity of SARS-CoV-2 is high, indicated by the worldwide spread of the disease in a very short period, many individuals displayed only subclinical infection, and some of them transmitted the disease to individuals who then developed a severe symptomatic infection. Furthermore, there are differences in the severity of infection across countries, which can be attributed to factors such as the emergence of viral mutations in a short period of time as well as to the immune responses to viral factors. Anti-viral immunity generally consists of neutralizing antibodies that block viral infection and cytotoxic CD8+ T cells that eliminate the virus-infected cells. There is compelling evidence for the role of neutralizing antibodies in protective immunity in SARS-CoV-2 infection. However, the role of CD4+ and CD8+ T cells after the viral entry is complex and warrants a comprehensive discussion. Here, we discuss the protection afforded by cellular immunity against initial infection and development of severe disease. The initial failure of cellular immunity to control the infection worsens the clinical outcomes and functional profiles that inflict tissue damage without effectively eliminating viral reservoirs, while robust T cell responses are associated with mild outcomes. We also discuss persistent long-lasting memory T cell-mediated protection after infection or vaccination, which is rather complicated as it may involve SARS-CoV-2-specific cytotoxic T lymphocytes or cross-reactivity with previously infected seasonal coronaviruses, which are largely related to HLA genotypes. In addition, cross-reactivity with mutant strains is also discussed. Lastly, we discuss appropriate measures to be taken against the disease for immunocompromised patients. In conclusion, we provide evidence and discuss the causal relationship between natural infection- or vaccine-mediated memory T cell immunity and severity of COVID-19. This review is expected to provide a basis to develop strategies for the next generation of T cell-focused vaccines and aid in ending the current pandemic.