Based on principles of chemical thermodynamics, the relationship between temperature and the Gibbs free energy of erosion products generated during the sulfate attack on cement concrete was deduced. The orientation of chemical reactions of sulfate attack on cement concrete was theoretically determined as well as the critical sulfate ion concentration and the formation conditions of erosion products. The phase composition, microstructure, crystal form, and morphology of erosion products before and after sulfate attack were investigated by environmental scanning electron microscope and energy spectrum analysis (ESEM-EDS) and X-ray diffraction (XRD). The results show that the effects of sulfate ion concentration and temperature on cement concrete sulfate attack are significant, and different influencing factors correlate with each other. The crystal transition temperature between the anhydrite and dihydrate gypsum is 42°C, and the corresponding concentration of sulfate ion is about 2.3 × 10−3 mol/L. Simultaneously, the crystal transition temperature between the thenardite and mirabilite is 32.4°C. Moreover, the theoretical upper limit temperature and sulfate ion lower limit concentration of thaumasite are 44°C and 0.0023 mol/L, respectively. The ESEM-EDS and XRD results imply that the chemical thermodynamics can be used to reveal the erosion mechanism of sulfate attack on cement concrete. The major erosion products of sulfate attack on cement concrete are rod-like ettringite with a larger slenderness ratio, plate-like gypsum, granular sulfate salt, incompletely corroded calcium hydroxide, and residual skeleton of calcium silicate hydrate. The sulfate attack has double effects on mechanical properties of specimens, which can affect the microstructure, phase composition, type, and morphology of erosion products.