The use of steel slag powder instead of filler to prepare asphalt mortar was beneficial to realize the effective utilization of steel slag and improve the performance of asphalt concrete. Nevertheless, the anti-aging properties of steel-slag powder–asphalt mortar need to be further enhanced. This study used antioxidants and UV absorbers in steel-slag powder–asphalt mortar to simultaneously improve its thermal-oxidation and UV-aging properties. The dosage of modifier was optimized by second-generation non-inferior sorting genetic algorithm. Fourier-Transform Infrared Spectroscopy, a dynamic shear rheometer and the heavy-metal-ion-leaching test were used to evaluate the characteristic functional groups, rheological properties and heavy-metal-toxicity characteristics of the steel-slag-powder-modified asphalt mortar, respectively. The results showed that there was a significant correlation between the amount of modifier and G*, δ, and the softening point. When the first peak appeared for G*, δ, and the softening point, the corresponding dosages of x1 were 2.15%, 1.0%, and 1.1%, respectively, while the corresponding dosage of x2 were 0.25%, 0.76%, and 0.38%, respectively. The optimal value of the modifier dosage x1 was 1.2% and x2 was 0.5% after weighing by the NSGA-II algorithm. The asphalt had a certain physical solid-sealing effect on the release of heavy-metal ions in the steel-slag powder. In addition, the asphalt structure was changed under the synergistic effect of oxygen and ultraviolet rays. Therefore, the risk of leaching heavy-metal ions was increased with the inferior asphalt-coating performance on the steel-slag powder.