Resulting from severe inflammation and cell destruction, COVID-19 patients could develop pulmonary fibrosis (PF), which remains in the convalescent stage. Nevertheless, how immune response participates in the pathogenesis of PF progression is not well defined. To investigate that question, 12 patients with severe COVID-19 were included in the study. Peripheral mononuclear cell (PBMC) samples were collected shortly after their admission and proceeded for single-cell RNA sequencing (scRNA-seq). After 14 days of discharge, the patients were revisited for chest CT scan. PF index (FI) was computed by AI-assisted CT images. Patients were categorized into FIhi and FIlo based on median of FI. By scRNA-seq analysis, our data demonstrated that frequency of CD4+ activated T cells and Treg cells were approximately 3-fold higher in FIhi patients compared with FIlo ones (p < 0.034 for all). By dissecting the differentially expressed genes, we found an overall downregulation of IFN-responsive genes (STAT1, IRF7, ISG15, ISG20, IFIs, and IFITMs) and S100s alarmins (S100A8, S100A9, S100A12, etc.) in all T-cell clusters, and cytotoxicity-related genes (GZMB, PRF1, and GNLY) in CTLs and γΎ T cells in the FIhi cohort, compared with FIlo subjects. The GSEA analysis illustrated decreased expression of genes enriched in IFN signaling, innate immune response, adaptive immune response in T cells, NK cells, and monocytes in FIhi patients compared with FIlo ones. In conclusion, these data indicated that the attenuated IFN-responsive genes and their related signaling pathways could be critical for PF progression in COVID-19 patients.