Phytoestrogens can alleviate some pathological processes related to nonalcoholic fatty liver disease (NAFLD). However, there are limited and contradictory studies on the relationships between phytoestrogens (especially single phytoestrogen) and NAFLD. The purpose of this study was to explore the relationships between urinary phytoestrogen concentrations and NAFLD in American adults. This cross-sectional study used the data of the National Health and Nutrition Examination Survey from 1999 to 2010, and 2294 adults were finally enrolled in this study. The concentrations of phytoestrogens were measured in urine samples, and urinary phytoestrogens were divided into tertiles according to the concentration distributions. The diagnosis of NAFLD was determined by the United States fatty liver index. The main analysis used a multivariate logistic regression model. The fully adjusted models included gender, age, race, education, marriage, poverty, body mass index, waist circumference, smoking, diabetes, hypertension, total cholesterol, high-density lipoprotein cholesterol, triglycerides, and other five phytoestrogens. In the fully adjusted model, the urinary enterolactone (ENL) concentration was negatively correlated with NAFLD (OR of Tertile 3 : 0.48, 95% CI 0.25–0.94). When stratified by age and gender, the urinary ENL concentration was negatively correlated with NAFLD in males aged 40–59 years (OR of Tertile 3 : 0.08, 95% CI 0.01–0.82), while the urinary equol concentration was positively correlated with NAFLD in such population (OR of Tertile 3 : 4.27, 95% CI 1.02–17.85). In addition, a negative correlation between enterodiol (END) concentration and NAFLD was observed in males aged 60 years or over (OR of Tertile 2 : 0.18, 95% CI 0.05–0.69). Collectively, in middle-aged males, urinary ENL may be associated with a lower risk of NAFLD, while urinary equol may be related to a higher risk. In addition, urinary END has a possible relationship with a reduced risk of NAFLD in elder males. Definitely, clinical randomized controlled trials are needed to further verify the conclusions.