Obesity has become a severe public health challenge globally. The present study aimed to identify separate and interactive dietary, genetic, and other factors that increase the risk of obesity as measured by body fat (BF) mass. We utilized a genome-wide association study to identify genetic variants associated with high fat mass (obesity; n = 10,502) and combined them to generate polygenic risk scores (PRS) of genetic variants interacting with each other in adults aged over 40 while excluding body-fat-related diseases in a city-hospital-based cohort (n = 53,828). It was validated in Ansan/Ansung plus rural cohorts (n = 13,007). We then evaluated dietary and lifestyle factors in subjects to assess what factors might help overcome a genetic propensity for higher BF. The three-SNP model included brain-derived neurotrophic factor (BDNF)_rs6265, fat-mass- and obesity-associated protein (FTO)_rs1421085, and SEC16B_rs509325. The genes with the minor alleles of ADCY3_rs6545790 and BAIAP2_rs35867081 increased their gene expression in the visceral and subcutaneous adipocytes, but their gene expression decreased in the hypothalamus in eQTL analysis. In the three-SNP model, the PRS was associated with BF mass by 1.408 and 1.396 times after adjusting covariates 1 (age, gender, survey year, residence area, education, and income) and 2 (covariates in model 1 plus energy intake, alcohol intake, regular exercise, and smoking status), respectively. However, when separating subjects by PRS of the three-SNP model, a plant-based diet was the most significant factor associated with low BF, followed by high-protein diets and lower energy intakes. They could offset the effects of high genetic risk for high BF. In conclusion, modulating nutrient intakes might overcome a high genetic risk for obesity. Dietary choices favoring more plant-based and higher-protein foods might help prevent increased BF in Asians and potentially people of other ethnicities with high polygenetic risk scores.