In this study, an acidic polysaccharide from Codonopsis pilosula Nannf. var. modesta (Nannf.) L. T. Shen (WCP-I) and its main fragment, WCP-Ia, obtained after pectinase digestion, were structurally elucidated and found to consist of a rhamnogalacturonan I (RG-I) region containing both arabinogalactan type I (AG-I) and type II (AG-II) as sidechains. They both expressed immunomodulating activity against Peyer’s patch cells. Endo-1,4-β-galactanase degradation gave a decrease of interleukine 6 (IL-6) production compared with native WCP-I and WCP-Ia, but exo-α-l-arabinofuranosidase digestion showed no changes in activity. This demonstrated that the stimulation activity partly disappeared with removal of β-d-(1→4)-galactan chains, proving that the AG-I side chain plays an important role in immunoregulation activity. WCP-Ia had a better promotion effect than WCP-I in vivo, shown through an increased spleen index, higher concentrations of IL-6, transforming growth factor-β (TGF-β), and tumor necrosis factor-α (TNF-α) in serum, and a slight increment in the secretory immunoglobulin A (sIgA) and CD4+/CD8+ T lymphocyte ratio. These results suggest that β-d-(1→4)-galactan-containing chains in WCP-I play an essential role in the expression of immunomodulating activity. Combining all the results in this and previous studies, the intestinal immune system might be the target site of WCP-Ia.