Recycling coal gangue as aggregate to produce concrete in situ is the most effective way to solve the problem of deposited coal gangue in mines. Nevertheless, the mine environment underground is rich in sulfate ions, posing a threat to the durability of coal gangue concrete (CGC). Hence, the degradation process of sulfate-attacked CGC is investigated. A series of tests is performed to evaluate its variation law of mass, dynamic elastic modulus, compressive strength and sulfate ion distribution. Meanwhile, the microstructure and phases of sulfate-attacked CGC are identified by scanning electron microscopy, X-ray diffraction and thermogravimetric analysis. The results indicate that the residual compressive strength ratio of CGC is higher than that of normal concrete after a 240 d sulfate attack, implying a superior sulfate resistance for CGC. Additionally, the higher the sulfate concentration, the more severe the degradation. Except for the secondary hydration of CGC itself, the diffused sulfate ions also react with Ca(OH)2, forming gypsum and ettringite; this plays a positive role in filling the pores at the early stage, whereas, at the later stage, the generated micro-cracks are detrimental to the performance of CGC. In particular, the proposed sulfate corrosion model elucidates the degradation mechanism of CGC exposed to a sulfate-rich environment.