Background: Betel nut chewing is associated with oral cancer, cardiovascular disease, liver cirrhosis, and hepatocellular carcinoma (HCC). The aim of this study was to explore the association of betel nut chewing with liver fibrosis in subjects with and without nonalcoholic fatty liver disease (NAFLD). Method: A total of 5967 subjects were enrolled. NAFLD was diagnosed with ultrasonography. Betel nut chewing was classified into non-chewing, ex-chewing, and current chewing, and cumulative dosages were calculated. The aspartate aminotransferase (AST)/platelet ratio index and NAFLD fibrosis scores (NFS) were calculated for evaluation of liver fibrosis. Results: NAFLD increased the associated risk of liver fibrosis in those with (odds ratio (OR): 5.51, 95% confidence interval (CI): 3.09–9.80) and without betel nut chewing (OR: 2.33, 95% CI: 1.64–3.29). In subjects without NAFLD, betel nut chewing was not associated with liver fibrosis (OR: 1.12, 95% CI: 0.44–2.86). In subjects with NAFLD, cumulative betel nut chewing and ex- and current chewing were positively associated with NFS and significant liver fibrosis. Conclusions: In subjects with NAFLD, betel nut chewing, even ex-chewing, was associated with a higher risk of liver fibrosis, where higher cumulative levels were found to increase the risk of significant liver fibrosis. However, the associated risk of liver fibrosis due to betel nut chewing was insignificant in subjects without NAFLD.