To figure out how does SARS-CoV-2 affect sperm parameters and what influencing factors affect the recovery of sperm quality after infection? We conducted a prospective cohort study and initially included 122 men with SARS-CoV-2 infection. The longest time to track semen quality after infection is 112 days and 58 eligible patients were included in our study eventually. We subsequently exploited a linear mixed-effects model to statistically analyze their semen parameters at different time points before and after SARS-CoV-2 infection. Semen parameters were significantly reduced after SARS-CoV-2 infection, including total sperm count (211 [147; 347] to 167 [65.0; 258], P < 0.001), sperm concentration (69.0 [38.8; 97.0] to 51.0 [25.5; 71.5], P < 0.001), total sperm motility (57.5 [52.3; 65.0] to 51.0 [38.5; 56.8], P < 0.001), progressive motility (50.0 [46.2; 58.0] to 45.0 [31.5; 52.8], P < 0.001). The parameters displayed the greatest diminution within 30 days after SARS-CoV-2 infection, gradually recovered thereafter, and exhibited no significant difference after 90 days compared with prior to COVID-19 infection. In addition, the patients in the group with a low-grade fever showed a declining tendency in semen parameters, but not to a significant degree, whereas those men with a moderate or high fever produced a significant drop in the same parameters. Semen parameters were significantly reduced after SARS-CoV-2 infection, and fever severity during SARS-CoV-2 infection may constitute the main influencing factor in reducing semen parameters in patients after recovery, but the effect is reversible and the semen parameters gradually return to normal with the realization of a new spermatogenic cycle.