Ceratophyllum demersum L. (CDL) is a traditional Chinese herb to treat many diseases, but research on its anti-diabetic activity is not available. In this research, the α-glucosidase inhibitory ability and phytochemical constituents of CDL extract were firstly studied. Optimal ultrasound-assisted extraction conditions for α-glucosidase inhibitors (AGIs) were optimized by single factor experiment and response surface methodology (RSM), which was confirmed as 70% methanol, liquid-to-solid ratio of 43 (mL/g), extraction time of 54 min, ultrasonic power of 350 W, and extraction temperature of 40 °C. The lowest IC50 value for α-glucosidase inhibition was 0.15 mg dried material/mL (mg DM/mL), which was much lower than that of acarbose (IC50 value of 0.64 mg DM/mL). In total, 80 compounds including 8 organic acids, 11 phenolic acids, 25 flavonoids, 21 fatty acids, and 15 others were identified or tentatively identified from CDL extract by HPLC-QTOF-MS/MS analysis. The results suggested that CDL could be a potential source of α-glucosidase inhibitors. It can also provide useful phytochemical information for research into other bioactivities.