Open stockpiling and the continual production of industrial solid wastes such as red mud (RM) and yellow phosphorus slag (YPS) have caused serious environmental pollution issues. Additionally, concrete prepared easily and with high strength is a widely applied building material. Therefore, replacing part or all of the cement for preparing concrete with RM and YPS will greatly reduce this kind of solid waste and, thus, decrease environmental pressures. This study investigated the best ratio for the replacement of concrete with RM and YPS, testing the mechanical properties as well as the morphology, material composition, and microporous structure of the interface transition zone (ITZ). The results showed for the concrete prepared with ordinary Portland cement replaced by 10.00 wt.% RM and 18 wt.% YPS, compared to ordinary Portland cement concrete, the compressive strength of concrete with basalt aggregate and dolomite aggregate increased by 25.04% and 27.27%, respectively, when the concrete was cured with steam for 28 days. Furthermore, it had a smaller average pore diameter and crystal size in the ITZ. The aggregate and matrix were more closely intertwined. This was because RM had a low cementitious activity and mainly had a filling effect when added to concrete, while the highly active silica in YPS could react with the Ca(OH)2 crystal (CH) produced from cement hydration to form calcium silicate hydrate (CSH) gel, improving the mechanical properties and microstructure of the concrete.