Two unreported heteropolysaccharides, denoted as YCJP–1 and YCJP–2, were isolated from the herbs of Chloranthus japonicus. YCJP–1 was a heteropolysaccharide composed of glucose, galactose, arabinose, mannose, rhamnose, and a minor proportion of uronic acids, with the molecular weight mainly distributed in the 74,475–228,443 Da range. YCJP–2 was mainly composed of glucose, mannose, and galactose, with the molecular weights ranging from 848 to 5810 Da. To further evaluate the anti-gastric cancer effects of C. japonicus, the inhibitory effects of the crude polysaccharide (YCJP) and the purified polysaccharides (YCJP–1 and YCJP–2) were determined using a CCK-8 assay and colon-forming assay on MGC-803 and AGS gastric cancer cell lines. Our results showed that YCJP, YCJP–1, and YCJP–2 possess prominent inhibitory effects on the proliferation of MGC-803 and AGS cells, and the AGS cell was more sensitive to YCJP, YCJP–1, and YCJP–2. Moreover, YCJP–2 demonstrated superior anti-gastric cancer effects compared to YCJP–1. This could potentially be attributed to YCJP–2’s higher glucose content and narrower molecular weight distribution.