The COVID-19 pandemic has significantly affected individuals’ physical and mental health, including that of immigrant women. This study aimed to evaluate the health-related quality of life (HRQoL), identify the demographic factors and awareness of the COVID-19 pandemic contributing to physical and mental health, and examine the risk factors associated with poor physical and mental health of Chinese women in Japan following the COVID-19 pandemic outbreak. Using an electronic questionnaire survey, we collected data including items on HRQoL, awareness of the COVID-19 pandemic, and demographic factors. One hundred and ninety-three participants were analyzed. Approximately 98.9% of them thought that COVID-19 affected their daily lives, and 97.4% had COVID-19 concerns. Married status (OR = 2.88, 95%CI [1.07, 7.72], p = 0.036), high concerns (OR = 3.99, 95%CI [1.46, 10.94], p = 0.007), and no concerns (OR = 8.75, 95%CI [1.17, 65.52], p = 0.035) about the COVID-19 pandemic were significantly associated with poor physical health. Unmarried status (OR = 2.83, 95%CI [1.20, 6.70], p = 0.018) and high COVID-19 concerns (OR = 2.17, 95%CI [1.04, 4.56], p = 0.040) were significantly associated with poor mental health. It is necessary to provide effective social support for Chinese women in Japan to improve their well-being, especially in terms of mental health.