Chitosan oligosaccharide (COS) is a bioactive compound derived from marine by-products. COS consumption has been demonstrated to lower the risk of diabetes. However, there are limited data on the inhibitory effect of low-molecular-weight COSs with different degrees of polymerization (DP) on α-glucosidase. This study investigates the α-glucosidase inhibitory activity of two low-molecular-weight COSs, i.e., S-TU-COS with DP2–4 and L-TU-COS with DP2–5, both of which have different molecular weight distributions. The inhibition constants of the inhibitors binding to free enzymes (Ki) and an enzyme–substrate complex (Kii) were investigated to elucidate the inhibitory mechanism of COSs with different chain lengths. The kinetic inhibition model of S-TU-COS showed non-completive inhibition results which are close to the uncompetitive inhibition results with Ki and Kii values of 3.34 mM and 2.94 mM, respectively. In contrast, L-TU-COS showed uncompetitive inhibition with a Kii value of 5.84 mM. With this behavior, the IC50 values of S-TU-COS and L-TU-COS decreased from 12.54 to 11.84 mM and 20.42 to 17.75 mM, respectively, with an increasing substrate concentration from 0.075 to 0.3 mM. This suggests that S-TU-COS is a more potent inhibitor, and the different DP of COS may cause significantly different inhibition (p < 0.05) on the α-glucosidase activity. This research may provide new insights into the production of a COS with a suitable profile for antidiabetic activity.