Phosphogypsum is a common industrial solid waste that faces the challenges of high stockpiling and low utilization rates. This study focuses on the mechanical properties and internal characteristics of cementitious materials with a high phosphogypsum content. Specifically, we examined the effects of varying amounts of ground granulated blast furnace slag (5–28%), fly ash (5–20%), and hydrated lime (0.5–2%) on the stress–strain curve, unconfined uniaxial compressive strength, and elastic modulus (E50) of these materials. The test results indicate that increasing the ground granulated blast furnace slag content can significantly enhance the mechanical properties of phosphogypsum-based cementitious materials. Additionally, increasing the fly ash content can have a similar beneficial effect with an appropriate amount of hydrated lime. Furthermore, microscopic analysis of the cementitious materials using a scanning electron microscope revealed that the high sulfate content in phosphogypsum leads to the formation of calcium aluminate as the main product. Concurrently, a continuous reaction of the raw materials contributes to the strength development of the cementitious materials over time. The results could provide a novel method for improving the reusing phosphogypsum amount in civil engineering materials.