Background: The association between poverty and health has been widely assessed. However, whether the association between poverty and health-related quality of life (HRQOL) holds among different groups is unknown. This study aimed to 1) assess the association between poverty and HRQOL among rural residents in China and 2) examine whether the association holds among different populations, thereby supporting policy-making and implementation. Methods: A multistage, stratified, random household survey was conducted with self-administered questionnaires. Matched samples were generated by the censored exact matching method to reduce selection bias between the poverty and comparison groups. We applied Tobit and ordinal logit regression models to evaluate the association between poverty and HRQOL measured by the EQ-5D-3 L among different groups. Results: The health utility score of the poverty group was 6.1% lower than that of comparison group (95% CI = − 0.085, − 0.037), with anxiety/depression being most common (95% CI = 1.220, 1.791). The association between poverty and HRQOL was significantly stronger among residents from central China, males, people who were middle-aged, elderly, highly educated, married, or widowed, those living far from healthcare facilities, and those without chronic disease. Male and highly educated subjects reported worse mobility, selfcare, usual activities, pain/discomfort and anxiety/depression dimensions than the other respondents. Middleaged (95% CI = 1.692, 2.851) and married respondents (95% CI = 1.692, 2.509) and respondents with chronic diseases (95% CI = 1.770, 2.849) were most affected in the anxiety/depression. Conclusions: The HRQOL of individuals living in poverty is lower than that of the general population, and the mental health dimension is most affected by poverty among respondents who are middle-aged or married and respondents with chronic diseases. The identification of populations that are more affected by poverty is critical to improve their HRQOL. Various associations have indicated the need for integrated policies and specific decision-making.