This cross-sectional study aimed to determine the associations between food groups and health-related quality of life (HRQoL). The data of 14,979 participants in the Korea National Health and Nutrition Examination Survey from 2016 and 2018 were examined. The HRQoL was assessed with EuroQol-5 Dimension. The 24-h recall test was used to examine the dietary intake of food groups. Males and females accounted for 13.79% and 21.62% of the low HRQoL groups. The males in the lowest tertile of the “vegetables” and “fish and shellfish” food groups were more likely to have a low HRQoL (odds ratio (OR), 95% confidence interval (CI) = 1.25 (1.05–1.65), 1.45 (1.12–1.89), respectively) than those in the highest tertile, whereas those in the lowest tertile of the “cereal and grain products” were less likely to have a low HRQoL (OR (95% CI) = 0.69 (0.52–0.91)). The females in the lowest tertile of the “vegetables” food group were more likely to have a low HRQoL (OR (95% CI) = 1.56 (1.17–3.01)) than those in the highest tertile. After adjusting for confounders indagated with not only dietary but also non-dietary factors such as stress, we found that low HRQoL was significantly associated with food groups of “vegetables”, “fish and shellfish”, and “cereal and grain products” among males and of “vegetables” among females.