Serious environmental pollution issues caused by the storage and exposure of large amounts of red mud (RM) and yellow phosphorus slag (YPS) have raised significant concerns. Red mud-yellow phosphorous slag-cement concrete (RM-YPS-CC) is prepared with 25% yellow phosphorus slag content (YPSC) and 10% red mud content (RMC) to replace a portion of the cement at the age of 28 days and was found in this study to satisfy the mechanical property requirements. More ettringite and portlandite were generated with the RM-YPS-CC than with the yellow phosphorous slag-cement concrete (YPS-CC). In addition, the cementitious materials were more interlaced, and there was more disorder in the crystals of the RM-YPS-CC, which formed a more complex spatial structure than the YPS-CC did. Without RM, the initial cracking strength on the surface of the concrete was 5–6 MPa, the maximum crack width was 3.96 mm, and the crack number was 8. However, the cracking strength was 26.5–27 MPa with RMC5, the maximum crack width was 0.66 mm with RMC15, and the crack number was 3 with RMC15. Moreover, studies using the digital image correlation (DIC) method indicated that the displacement distribution and evolution of the first crack area changed quickly at 10 MPa in either horizontal or vertical direction, and a similar trend was maintained from 10 MPa to 27.1 MPa for the YPS-CC. However, with a small distribution and evolution of horizontal or vertical displacement from 5 MPa to 25 MPa, the evolution would change rapidly when reaching 30 MPa for RM-YPS-CC. This study aims to provide new insights into the wide application of YPS and RM for saving energy and reducing emissions and to develop a new method to study the fracture behavior of concrete.