The high belite sulfoaluminate cement (HBSAC) containing CaSO4, and without CaSO4, based on solid wastes were successfully prepared from petroleum coke desulfurization slag (PCDS), fly ash (FA), carbide slag (CS), and bauxite (BX). The mineral composition of clinkers after different calcination history were investigated by X-ray fluorescence (XRF), X-ray diffraction (XRD)/Quantitative X-ray diffraction (QXRD), and scanning electron microscopy (SEM), so as to determine the calcination temperatures. The difference between residual CaSO4 and dihydrate gypsum (DG) and the optimal content of residual CaSO4 were discussed by studying the properties of HBSAC. The results revealed that the residual CaSO4 in clinker could replace DG to participate in hydration, and showed some advantages in strength and early hydration heat, but meanwhile increased the water requirement of normal consistency and hydration heat at 72 h, and prolonged the setting time. With the increase of residual CaSO4 content in clinker, the lower limit temperature of clinker formation gradually increased, and the crystal size of clinker minerals became finer and the boundary between crystals became more blurred. However, the optimal calcination temperature (1300 °C) of HBSAC clinker did not change. Considering the effect of residual CaSO4 content on the water requirement of normal consistency, setting time, hydration heat, strength, and hydration products, the optimal design content of residual CaSO4 in HBSAC clinker based on solid wastes, such as PCDS and FA, was 15%.