Industrial byproduct gypsum (BPG) is a secondary product that is mainly composed of calcium sulfate discharged during industrial production. BPG primarily consists of desulfurized gypsum, phosphogypsum, and titanium gypsum, which account for 88% of the total BPG in China. The large-scale utilization of these three types of solid waste is crucial for the safe disposal of BPG. BPG contains various impurities and harmful elements, limiting its applications. The continuous accumulation of BPG poses a serious threat to the safety of the environment. Based on a literature review (2021–2023), it was found that 52% of BPG is used in the preparation of cementitious materials, and the addition of BPG results in an average improvement of 7–30% in the mechanical properties of cementitious materials. Moreover, BPG has a positive impact on the immobilization of hazardous elements in raw materials. Therefore, the utilization of BPG in cementitious materials is beneficial for its large-scale disposal. This study primarily reviews the effects and mechanisms of BPG on the mechanical properties of cementitious materials and the solidification of hazardous elements. Most importantly, the review reveals that BPG positively influences the hydration activity of silica–alumina-based solid waste (such as steel slag and blast furnace slag) and alkaline solid waste (such as carbide slag and red mud). This improves the proportion of solid waste in cement and reduces production costs and carbon emissions. Finally, this article summarizes and proposes the application of BPG in cementitious materials. The application of BPG + silica–alumina solid waste + alkaline solid-waste-based cementitious materials is expected to realize a new type of green ecological chain for the joint utilization of multiple industrial solid wastes and to promote the low-carbon sustainable development of industrial clusters.