Hepatocellular carcinoma (HCC) is one of the most common malignancies. The currently available clinical drugs for HCC frequently cause serious side effects and the treatment outcomes are unsatisfactory. It is urgent to develop effective drugs with high selectivity and low adverse effects for HCC. Metabolites produced by microorganisms have shown great potential in the development of therapeutic agents for HCC. In our study, the EtOAc extract of the strain Aspergillus fumigatus YB4-17 exhibited significant cytotoxicity towards the HCC HepG2 cells at 10 μg/mL. Various column chromatographic separations of the extract afforded seven polyketides (1–7), including a new diphenyl ether derivative (1), along with fourteen known alkaloids (8–21). The structure elucidation was conducted via NMR spectroscopic data and MS data analysis. The absolute configuration of compound 11 was confirmed by comparing experimental and calculated electronic circular dichroism spectrum for the first time. The biological evaluation of these metabolites revealed that compound 11 selectively inhibited the proliferation of HCC HepG2 cells with negligible toxicity to normal cells. Mechanism study indicated that compound 11 induced apoptosis and paraptosis in HepG2 cells, providing a novel therapeutic perspective for the treatment of hepatocellular carcinoma.