SARS-CoV-2-specific memory T cells that cross-react with common cold coronaviruses (CCCs) are present in both healthy donors and COVID-19 patients. However, whether these cross-reactive T cells play a role in COVID-19 pathogenesis versus protection remain to be fully elucidated. In this study, we characterized cross-reactive SARS-CoV-2-specific CD4+ and CD8+ T cells, targeting genome-wide conserved epitopes in a cohort of 147 non-vaccinated COVID-19 patients, divided into six groups based on the degrees of disease severity. We compared the frequency, phenotype, and function of these SARS-CoV-2-specific CD4+ and CD8+ T cells between severely ill and asymptomatic COVID-19 patients and correlated this with alpha-CCCs and beta-CCCs co-infection status. Compared with asymptomatic COVID-19 patients, the severely ill COVID-19 patients and patients with fatal outcomes: (i) Presented a broad leukocytosis and a broad CD4+ and CD8+ T cell lymphopenia; (ii) Developed low frequencies of functional IFN-gamma-producing CD134+CD138+CD4+ and CD134+CD138+CD8+ T cells directed toward conserved epitopes from structural, non-structural and regulatory SARS-CoV-2 proteins; (iii) Displayed high frequencies of SARS-CoV-2-specific functionally exhausted PD-1+TIM3+TIGIT+CTLA4+CD4+ and PD-1+TIM3+TIGIT+CTLA4+CD8+ T cells; and (iv) Displayed similar frequencies of co-infections with beta-CCCs strains but significantly fewer co-infections with alpha-CCCs strains. Interestingly, the cross-reactive SARS-CoV-2 epitopes that recalled the strongest CD4+ and CD8+ T cell responses in unexposed healthy donors (HD) were the most strongly associated with better disease outcome seen in asymptomatic COVID-19 patients. Our results demonstrate that, the critically ill COVID-19 patients displayed fewer co-infection with alpha-CCCs strain, presented broad T cell lymphopenia and higher frequencies of cross reactive exhausted SARS-CoV-2-specific CD4+ and CD8+ T cells. In contrast, the asymptomatic COVID-19 patients, appeared to present more co-infections with alpha-CCCs strains, associated with higher frequencies of functional cross-reactive SARS-CoV-2-specific CD4+ and CD8+ T cells. These findings support the development of broadly protective, T-cell-based, multi-antigen universal pan-Coronavirus vaccines.