The use of bulk solid-waste iron tailing (IOT), red mud (RM), and oyster shells to prepare cemented paste backfill (CPB) can effectively solve the ecological problems caused by industrial solid waste storage and improve the utilization rate of such materials. In this study, a new type of CPB was prepared by partially replacing slag with RM, with calcined oyster shell (COS) as the alkaline activator and IOT as aggregate. The central composite design (CCD) method was used to design experiments to predict the effects of the COS dosage, RM substitution rate, solid mass, and aggregate–binder ratio using 28-dUCS, slump, and the cost of CPB. In this way, a regression model was established. The quantum genetic algorithm (QGA) was used to optimize the regression model, and X-ray diffraction (XRD), Fourier transform infrared (FTIR), scanning electron microscope (SEM), and energy dispersive spectroscopy (EDS) microscopic tests are performed on CPB samples of different ages with the optimal mix ratio. The results showed that COS is a highly active alkaline substance that provides an alkaline environment for polymerization reactions. In the alkaline medium, the hematite and goethite in RM and quartz in IOT gradually dissolved and participated in the process of polymerization. The main polymerization products of the CPB samples are calcium–silicate–hydrogel (C–S–H), calcium–aluminosilicate–hydrogel (C–A–S–H), and aluminosilicate crystals such as quartz, albite, and foshagite. These products are intertwined and filled in the internal pores of the CPB, enabling the pore contents to decrease and the interiors of the CPB samples to gradually connect into a whole. In this way, the compressive strength is increased.