Panax quinquefolius (North American ginseng, NAG) is a popular medicinal plant used widely in traditional medicine. NAG products are currently available in various forms such as roots, extracts, nutraceuticals, dietary supplements, energy drinks, etc. NAG polysaccharides are recognized as one of the major bioactive ingredients. However, most NAG reviews are focused on ginsenosides with little information on polysaccharides. NAG polysaccharides have demonstrated a therapeutic activity in numerous studies, in which many of the bioactivities involve regulation of the immune response. The purpose of this review is to summarize the structural features and the immunomodulatory properties of crude, partially purified, and pure polysaccharides isolated from NAG. Receptors of the innate immune system that potentially bind to NAG polysaccharides and the respective signal transduction pathways initiated by these compounds are discussed. Major challenges, recent innovations, and future directions in NAG polysaccharide research are also summarized.