The cement industry could potentially consume large amounts of solid industrial waste in order to improve its sustainability. The suitability of selected quarry and mine waste as secondary raw materials (SRM) was examined for the sustainable production of cement following the concept of a circular economy. The chemical, mineralogical, and radiological characterization of SRM was conducted in this study. Its potential use in low-carbon and low-energy belite-sulfoaluminate cement was investigated by incorporating the examined SRM into cement clinker. Various characterization methods were used to characterize the cement, including X-ray powder diffraction (XRD), thermal analysis (DTA/TG), and isothermal calorimetry. Depending on the chemical composition of the waste, lower or higher amounts were allowed to be incorporated into the raw clinker mixture for a targeted clinker phase composition. Among the samples, differences were observed in the phase composition of synthesized clinkers, which slightly influenced the reactivity of the cement but did not significantly change the compressive strength of the final product.