Owing to the COVID-19 pandemic, classes and club activities in Japanese universities have been canceled; this may increase students' perceived stress and adversely affect their quality of life. This study investigated the factors that influence Japanese university students' quality of life during the pandemic. An online questionnaire collating data related to demographic characteristics, the perceived stress scale (PSS), sense of coherence (SOC), number of social supports, and quality of life (Short Form Health Survey; SF-8) was distributed to university students. Participants were divided into two groups: those who entered university before (seniors) and after (juniors) the onset of the pandemic. Their scores on the two sub-domains (physical and mental) of the SF-8 were evaluated. Multiple regression analysis was performed to identify factors associated with the composite mental summary of the SF-8. Regression analysis indicated that the predictor model of the composite mental summary differed between juniors and seniors. Among seniors, the composite mental summary was significantly indicated by the composite physical summary (β = 0.549, p < 0.0001) and PSS (β = 0.422, p < 0.0001). Among juniors, it was significantly indicated by the composite physical summary (β = 0.531, p < 0.0001), PSS (β = 0.390, p < 0.0001), and number of social supports (β = −0.148, p = 0.006). The factors associated with quality of life differed between seniors and juniors. Universities must provide opportunities for students to find more friends, especially for juniors who have limited socialization opportunities owing to the pandemic.