ImportanceThe coronavirus disease 2019 (COVID-19) pandemic may have a negative impact on mental health of the population, leading to higher suicide rates, in many countries. However, little is known about risk factors associated with worsened mental health during the COVID-19 pandemic.ObjectiveTo investigate the factors associated with serious psychological distress (SPD) during the COVID-19 pandemic in Japan.Design, Setting, and ParticipantsA cross-sectional study using a large-scale internet survey conducted between August 25 and September 30, 2020, in Japan.ExposuresDemographics (age, gender, marital status, family composition, and caregiving burden), socio-economic status (income level, employment type, educational attainment), the experience of domestic violence (DV), the state of emergency, fear of COVID-19, and stigma related to COVID-19.Main Outcomes and MeasuresPrevalence of SPD, defined as Kessler 6 scale score ≥13.ResultsAmong 25,482 individuals included in this study, 2,556 (10%) met the criteria of SPD. Overall, women (adjusted odds ratio [aOR] 1.59; 95%CI, 1.17–2.16; P=0.003), ages 15–29 (aOR compared with ages 45–59, 2.35; 95%CI, 1.64–3.38; P<0.001), low income (aOR compared with intermediate income, 1.70; 95%CI, 1.16–2.49; P=0.007), providing caregiving to family members (aOR, 5.48; 95%CI, 3.51–8.56; P<0.001), experiencing DV (aOR, 5.72; 95%CI, 3.81–8.59; P<0.001), and fear of COVID-19 (aOR, 1.96; 95%CI, 1.55–2.48; P<0.001) were associated with a higher rate of SPD. Among women aged 15–29 years, who experienced the highest rate of SPD, caregiving, DV, fear of COVID-19, and COVID-19-related stigma were associated with a higher rate of SPD; whereas economic situation (income level and employment type) and social isolation (marital status) were not associated with the prevalence of SPD.Conclusions and RelevanceEconomic situation, caregiving burden, DV, and fear of COVID-19 were independently associated with SPD during the COVID-19 pandemic. Among young women—who have a higher risk of suicide during the COVID-19 pandemic in Japan—similar factors, except economic situation, were associated with a higher rate of SPD. Targeted interventions based on age and gender may be more effective in mitigating the negative impact of the COVID-19 pandemic on the population’s mental health.