Damage to concrete structures with gypsum-contaminated aggregate occurs frequently. Aggregates in much of the southern part of China are contaminated with gypsum. Therefore, in this study, the effects of using different quantities of gypsum-contaminated aggregate on the expansion and compressive strength of concrete were investigated over a period of one year. Two groups of concrete were designed with the gypsum-contaminated aggregate containing different parts of fine and coarse aggregate, respectively. The SO3 contents were 0%, 0.5%, 1%, 1.5%, 3%, 5%, and 7% by weight of aggregate. X-ray diffraction (XRD), thermogravimetry (TG), and differential scanning calorimetry (DSC) were used to analyze the change in mineral composition over time. The microstructure was also studied by scanning electron microscopy (SEM) and energy dispersive spectrometry (EDS). The results showed that significant expansion and great loss in compressive strength did not occur in concrete if the content of SO3 lay below 1.5% and 3% in fine and coarse aggregates, respectively. The concentration of sulfate ions in concrete was not enough to form new a phase of gypsum. During the process of internal sulfate attack, the content of gypsum decreased and the content of ettringite increased. Ettringite was the main reason for the expansion damage of concrete. Additionally, the fracture mode of internal sulfate attack on concrete was the crack extension from gypsum to paste; finally, the aggregate separated from the paste.