Inhibition of glycogen synthase kinase 3β (GSK-3β) is considered to be the central therapeutic approach against Alzheimer’s disease (AD). In the present study, boiled water extracts of the Kangen-karyu (KK) herbal mixture and its constituents were screened for GSK-3β inhibitory activity. KK is used in traditional Kampo and Chinese medicines for improving cognitive function. The GSK-3β inhibition potential was evaluated by using the Kinase-Glo luminescent kinase assay platform. Furthermore, enzyme kinetics and in silico modeling were performed by using AutoDockTools to demonstrate the mechanism of enzyme inhibition. KK extract significantly inhibited GSK-3β in a concentration-dependent manner (IC50: 17.05 ± 1.14 μg/mL) when compared with the reference drug luteolin (IC50: 2.18 ± 0.13 μM). Among the six components of KK, extracts of Cyperi Rhizoma and Salviae Miltiorrhizae Radix significantly inhibited GSK-3β with IC50 values of 20.68 ± 2.50 and 7.77 ± 1.38 μg/mL, respectively. Among the constituents of the roots of S. miltiorrhiza water extract, rosmarinic acid, magnesium lithospermate B, salvianolic acid A, salvianolic acid B, and salvianolic acid C inhibited GSK-3β with IC50 values ranging from 6.97 to 135.5 μM. Salvianolic acid B was found to be an ATP-competitive inhibitor of GSK-3β and showed the lowest IC50 value (6.97 ± 0.96 µM). In silico modeling suggested a mechanism of action by which the hydrophobic, π–cation, and hydrophilic interactions of salvianolic acid B at ATP and substrate sites are critical for the observed GSK-3β inhibition. Therefore, one of the mechanisms of action of KK against AD may be the inhibition of GSK-3β and one of the active components of KK is the root of S. miltiorrhiza and its constituents: rosmarinic acid, magnesium lithospermate B, and salvianolic acids A, B, and C. Our results demonstrate the pharmacological basis for the use of KK against AD.