As two kinds of solid wastes, waste tires and steel slag have caused serious threats to the environment. Both waste tire rubber (WTR) and steel slag powder (SSP) can improve the performance of asphalt, while the performance indexes and modification mechanism of modified asphalt are not clear. In this paper, asphalt modified with SSP and WTR was prepared, and its performance was evaluated. The physical properties of asphalt modified with SSP and WTR, including penetration, the softening point, and viscosity, were investigated. Furthermore, high-temperature performance, fatigue resistance, low-temperature performance, and blending mechanism of asphalt modified with SSP and WTR were tested with a dynamic shear rheometer (DSR), bending beam rheometer (BBR), and Fourier transform infrared spectrometer (FTIR). The results showed that with the same content of WTR and SSP, WTR reveals a more significant modification effect on physical properties, fatigue, and low-temperature performance of base asphalt than SSP. The anti-rutting performance of SSP-modified asphalt is better than that of WTR-modified asphalt at 30~42 °C, and the anti-rutting performance of WTR-modified asphalt is better than that of SSP-modified asphalt at 42~80 °C. When the total content of WTR and SSP is the same, the physical properties, high-temperature resistance, fatigue resistance, and low-temperature performance of the asphalt modified with WTR and SSP decrease with the decrease in the ratio of WTR and SSP, and their performance is between WTR-modified asphalt and SSP-modified asphalt. Infrared spectrum results verified that the preparation of WTR- and SSP-modified asphalt is mainly a physical blending process. Overall, this research is conducive to promoting the application of modified asphalt with WTR and SSP in the construction of high-standard pavement.