Clarifying dominant factors determining the immune heterogeneity from non-survivors to survivors is crucial for developing therapeutics and vaccines against COVID-19. The main difficulty is quantitatively analysing the multi-level clinical data, including viral dynamics, immune response and tissue damages. Here, we adopt a top-down modelling approach to quantify key functional aspects and their dynamical interplay in the battle between the virus and the immune system, yielding an accurate description of real-time clinical data involving hundreds of patients for the first time. The quantification of antiviral responses gives that, compared to antibodies, T cells play a more dominant role in virus clearance, especially for mild patients (96.5%). Moreover, the anti-inflammatory responses, namely the cytokine inhibition and tissue repair rates, also positively correlate with T cell number and are significantly suppressed in non-survivors. Simulations show that the lack of T cells can lead to more significant inflammation, proposing an explanation for the monotonic increase of COVID-19 mortality with age and higher mortality for males. We propose that T cells play a crucial role in the immunity against COVID-19, which provides a new direction–improvement of T cell number for advancing current prevention and treatment.