Idesia polycarpa Maxim. var. vestita Diels (I. polycarpa) is well known as an edible oil plant which contains abundant linoleic acid and polyphenols. The objective of this study was to maximize the by-product of defatted fruit of I. polycarpa. We found that the fraction D of ethyl acetate extract (EF-D) contained more polyphenols, which contribute to its strong antioxidant activity by antioxidant assays (DPPH, ABTS, and FRAP). Meanwhile, EF-D showed a significant lipid-lowering effect on oleic acid- (OA-) induced hepatic steatosis in HepG2 cells through enhancing antioxidant activity, reducing liver damage, and regulating lipid metabolism, antioxidant, and inflammation-related gene expression. The SOD and T-AOC levels significantly increased, but the levels of MDA, AST, and ALT decreased obviously when treated with EF-D. In general, EF-D improved the antioxidant enzyme activities and decreased the hepatic injury activities. Besides, treatment with EF-D for NAFLD influenced lipid metabolism and inflammation by activating PPARα which was associated with the increased expression of CPT1 and decreased expression of SCD, NF-κB, and IL-1. Moreover, EF-D improved the oxidative stress system through activation of the Nrf2 antioxidant signal pathways and upregulated its target genes of HO-1, NQO1, and GSTA2. The results highlighted the EF-D from the defatted fruit of I. polycarpa regarding lipid-lowering, proving it to be a potential drug resource of natural products for treating the nonalcoholic fatty liver disease (NAFLD).