Non-alcoholic fatty liver disease (NAFLD) is often thought of as clinically asymptomatic. However, many NAFLD patients complain of fatigue and low mood, which may affect their quality of life (QoL). This may create a barrier to weight loss and hinder the achievement of NAFLD therapy goals. Our study aimed to evaluate the QoL in NAFLD patients vs. healthy volunteers, and to analyze likely influencing factors. From March 2021 through December 2021, we enrolled 140 consecutive adult subjects (100 NAFLD patients and 40 controls). Overall, 95 patients with NAFLD and 37 controls were included in the final analysis. Fatty liver was diagnosed based on ultrasonographic findings. We employed 36-Item Short Form Health Survey (SF-36) to evaluate QoL, Hospital Anxiety and Depression Scale (HADS) to identify anxiety and/or depression, and Fatigue Assessment Scale (FAS) to measure fatigue. NAFLD patients had significantly lower physical component summary scores, as well as significantly higher HADS-D scores, compared with the control group (Mann-Whitney U criterion = 1140.0, p = 0.001 and U = 1294.5, p = 0.022, respectively). Likewise, fatigue was more common in NAFLD patients (χ2 = 4.008, p = 0.045). Impaired QoL was significantly associated with fatigue (FAS score ≥ 22, p < 0.001) and depression (HADS-D ≥ 8, p < 0.001). In conclusion, NAFLD patients had significantly poorer QoL vs. controls, in particular with respect to the physical component of health. Impaired QoL may be associated with fatigue and depression, and together they may interfere with increased physical activity and lifestyle modifications in patients with NAFLD.