Objectives Inflammatory Bowel Disease (IBD) is an autoimmune disease, and the gut microbiota has become a new therapeutic target. Herbal medicine (HM) has shown good efficacy in the clinical treatment of IBD; however, the synergistic actions of the dominant chemicals in HM decoctions are unclear. Methods In this study, we explored whether the complicated interconnections between HM and the gut microbiota could allow crosstalk between HM ingredients. Saponins and polysaccharides, i.e., the dominant chemicals in the Codonopsis pilosula Nannf (CPN) decoction, were investigated in a dextran sulfate sodium- (DSS-) induced mouse model. Bacterial 16S rRNA sequencing analyzed the change of gut microbiota structure and diversity. Gas chromatography (GC) determined the content of short-chain fatty acids (SCFAs) in feces. ELISA detected the expression of proinflammatory and anti-inflammatory cytokines associated with TH17/Treg balance. UPLC-QTOF-MS technology combined with PKsolver software analyzed the absorption of the highest exposure for monomeric compounds of CPN saponins in serum. The results indicated that CPN polysaccharides showed prebiotic-like effects in mice with DSS-induced colitis by simultaneously stimulating the growth of three important probiotics, i.e., Bifidobacterium spp., Lactobacillus spp., and Akkermansia spp., and inhibiting the growth of pathogenic bacteria, including Desulfovibrio spp., Alistipes spp., and Helicobacter spp. Moreover, CPN polysaccharides improved intestinal metabolism, enhanced the production of short-chain fatty acids, upregulated the expression of anti-inflammatory cytokines and downregulated the secretion of proinflammatory cytokines correlated with Th17/Treg balance, promoted the absorption of certain CPN saponins in the serum, and stimulated recovery of the holistic gut microbiota. Conclusion CPN polysaccharides have the good prebiotic properties and shown good application prospects in the prevention and treatment of acute colitis. These findings provide insights into the specific bacteria responsible for active, inactive biotransformation of HM ingredients and those that are altered by HM administration.