Due to the low content of silicon and aluminum in red mud and the low reaction activity of red mud, when it was used to prepare composite cementitious materials, it was necessary to assist other aluminosilicates and improve their activity by certain methods. In this study, it was proposed to add slag to increase the percentage of silicon and aluminum in the system, and to improve the reactivity of the system through the activation effect of sulfate in phosphogypsum. The effects of slag and phosphogypsum contents on the mechanical properties and microstructures of composite cementitious materials were studied. X-ray diffraction analysis (XRD), thermogravimetric analysis (TG-DTG), and scanning electron microscopy (SEM) were used to analyze the effects of slag and phosphogypsum contents on the hydration products, microstructure, and strength formation mechanism of composite cementitious materials. The results show that with the increase of slag, the strength of the composite cementitious material increases gradually. When the slag content is 50%, the 28-day compressive strength reaches a maximum of about 14 MPa. Compared with the composite material without phosphogypsum, the composite cementitious material with 10–20% phosphogypsum showed higher strength properties, in which the 28-day compressive strength exceeds 24 MPa. The main reason for this is that the sulfate in phosphogypsum can cause the composite cementitious material to generate a large amount of ettringite and accelerate the dissolution of red mud and slag, increasing the release of aluminates, silicates, and Ca2+ to form more C-(A)-S-H and ettringite. In addition, a large amount of C-(A)-S-H makes ettringite and unreacted particles combine into a uniform and compact structure, thus improving the strength. When the content of phosphogypsum exceeds 40%, the 28-day compressive strength of the composite cementitious material drops below 12 MPa due to the presence of fewer hydration products and the expansion of ettringite.