Diabetes mellitus (DM) is a common chronic metabolic disease worldwide. The disturbance of the gut microbiota has a complex influence on the development of DM. Polysaccharides are one type of the most important natural components with anti-diabetic effects. Gut microbiota can participate in the fermentation of polysaccharides, and through this, polysaccharides regulate the gut microbiota and improve DM. This review begins by a summary of the sources, anti-diabetic effects and the gut microbiota regulation functions of natural polysaccharides. Then, the mechanisms of polysaccharides in regulating the gut microbiota to exert anti-diabetic effects and the structure–activity relationship are summarized. It is found that polysaccharides from plants, fungi, and marine organisms show great hypoglycemic activities and the gut microbiota regulation functions. The mechanisms mainly include repairing the gut burrier, reshaping gut microbiota composition, changing the metabolites, regulating anti-inflammatory activity and immune function, and regulating the signal pathways. Structural characteristics of polysaccharides, such as monosaccharide composition, molecular weight, and type of glycosidic linkage, show great influence on the anti-diabetic activity of polysaccharides. This review provides a reference for the exploration and development of the anti-diabetic effects of polysaccharides.