Understanding individual perception of Quality of Life (QoL) can help combat social and health inequalities. We aimed to identify factors associated with Low Perceived Quality of Life (LPQoL) in 295 adults and older adults with food security and food insecurity, in the city of Natal, Brazil. A cross-sectional study was conducted from June to December 2019, with collection of data on socioeconomic demographic status, lifestyle information, non-communicable diseases (NCDs) and risk factors, emotional disorders, food (in) security and quality of life. To assess food insecurity, the Brazilian Scale of Food Insecurity—EBIA was used, and the WHOQOL-Bref questionnaire was used to assess quality of life. Poisson's Regression was used to verify associations between variables and LPQoL, stratifying the sample into food secure (FS) and food insecure (FI) groups. In the FI group, being overweight, older adult, having no partner, drinking alcoholic beverages twice a week or more, and not having daily availability of water were associated with LPQoL, and in the FS group, having diabetes, monthly family income in the 1st and 2nd tertiles, and never having studied was associated with LPQoL. Reporting emotional disorders and sleeping < 7 h/day were associated with LPQoL in both FI and FS groups. LPQoL was associated with the occurrence of NCDs and risk factors, and emotional disorders, regardless of the food security measure. However, the lack of adequate access to water highlights the social vulnerability of the FI group.