With the development of nanotechnology, reduced graphene oxide (rGO) has been used to improve the flexural strength of geopolymers. However, the reinforcing mechanism of rGO nanosheets on the flexural strength of geopolymers remains unclear. Here, this reinforcing mechanism was investigated from the perspectives of hydration and chemical composition. The effect of the reduction degree on rGO-reinforced geopolymers was also studied using isothermal calorimetry (IC), X-ray diffraction (XRD), and nuclear magnetic resonance (NMR) tests. Results show that the hydration degree and flexural strength of geopolymers effectively increase due to rGO addition. After alkali reduction at a temperature of 60 °C, rGO nanosheets have maximum reinforcement on the flexural strength of geopolymers with an increment of 51.2%. It is attributed to the promotion of slag hydration, as well as the simultaneous formation of calcium silicate hydrate with low Ca/Si ratio (C-S-H(I)) and calcium aluminosilicate hydrate (C-A-S-H) phases due to the inhibiting effect of rGO nanosheets on Al substitution on the end-of-chain silicates of C-S-H and C-A-S-H gels. In addition, different reduction degrees have almost no effect on the chemical composition of rGO-reinforced geopolymers, while excessive reduction impairs the improving effect of rGO nanosheets on the hydration process and flexural strength of geopolymers due to significant structural defects.