The aftereffects of the severe 2016 Kumamoto earthquake were complicated by the COVID-19 pandemic. This study aimed to identify mental health problems and related factors among survivors five years after the earthquake and clarify its long-term effects. A cross-sectional survey was conducted in 2020 among 19,212 survivors affected by the earthquake who moved from temporary to permanent housing. We analysed 8966 respondents (5135 women, 3831 men; mean age 62.25 ± 17.29 years). Logistic regression analysis was conducted to examine associations between mental health problems and socioeconomic factors. Prevalence rates of psychological distress, insomnia, and probable post-traumatic stress disorder were 11.9%, 35.2%, and 4.1%, respectively. Female gender (OR = 1.33, 95% CI = 1.13–1.57; OR = 1.21, 95% CI = 1.08–1.34; OR = 1.81, 95% CI = 1.41–2.32), public housing (OR = 2.14, 95% CI = 1.63–2.83; OR = 1.54, 95% CI = 1.26–1.88; OR = 2.41, 95% CI = 1.62–3.58), loneliness (OR = 9.08, 95% CI = 7.71–10.70; OR = 5.55, 95% CI = 4.90–6.30; OR = 3.52, 95% CI = 2.77–4.49), COVID-19-induced activity reduction (OR = 1.41, 95% CI = 1.19–1.66; OR = 1.86, 95% CI = 1.68–2.07; OR = 1.80, 95% CI = 1.40–2.31), and COVID-19-induced income reduction (OR = 1.33, 95% CI = 1.12–1.57; OR = 1.43, 95% CI = 1.28–1.59; OR = 1.92, 95% CI = 1.51–2.43) were significantly associated with mental health problems. These results suggest that gender, current housing, loneliness, and COVID-19 affected the survivors’ mental health during recovery.