Mexican tea (Dysphania ambrosioides (L.) Mosyakin & Clemants) is rich in phenolic acids and flavonoids and could be a potential medicinal herb that can be used for prevention of human hepatocellular carcinoma. The objective of this study was to elaborate the possible mechanism for the prevention or treatment of hepatocellular carcinoma using Mexican tea, and to provide new avenues for the utilization of the invasive plant. In this study, the D. ambrosioides seed extracts (CSE) were analyzed by gas chromatography-mass spectrometry, and the effects of CSE on proliferation, migration, invasion, and gene expression of SMMC-7721 cells were investigated. Eight compounds were identified in CSE, and the compound with the highest content was ascaridole (25.82 %). The proliferation was significantly inhibited by CSE (p < 0.05), and IC 50 values were 0.587 g/L, 0.360 g/L, and 0.361 g/L at 24 h, 36 h, and 48 h, respectively. Migration and invasion were significantly inhibited (p < 0.05). The network pharmacology and transcriptome analysis indicated that 2-hydroxy-2,6,6-trimethylbicyclo[3.1.1]heptan-3-one, cis-11-eicosenoic acid and 2-ethylcyclohexanone might be the active compounds. Transcriptome analysis indicated that the Wnt signaling pathway, which is related to migration and invasion, was significantly altered; this was verified by western blot assay. The expression of wnt11, lef1 and mmp7 genes in SMMC-7721 cells was significantly down-regulated (p < 0.05), while gsk-3β was significantly up-regulated (p < 0.05). These results indicate that CSE inhibits the invasion and migration of SMMC-7721 cells in hepatocellular carcinoma through the Wnt signaling pathway.