The polyphenol–starch complex has become a hot research topic since it is evident that this modification method can alter the physicochemical properties of starch as well as improve its nutritional value. This work aimed to evaluate the effect of ginger polyphenol gingerols (GNs) and shogaols (SNs) on the structure of starch with different amylose content (WCS, CS, G56, G80). Textural and rheological results indicated that GNs and SNs had more pronounced inhibitory retrogradation effects for relative low-level amylose starches (WCS and CS) compared to relative high-level amylose starches (G56 and G80). GNs and SNs improved the freeze–thaw stability of starch gels. FT-IR and XRD results revealed that GNs and SNs decreased the (short- and long-range) ordered structure of starches through a non-covalent interaction. Moreover, DSC results proved that the gelatinisation temperature of CS/G56/G80 significantly increased, and the enthalpy (ΔH) decreased by the incorporation of GNs and SNs. Overall, this in-depth study is beneficial in providing valuable pathways for starch–polyphenol interactions to improve the quality of starchy foods.