The continuous growth of industrial solid waste production has generated many environmental problems. We evaluated the potential of industrial solid waste as a substitute filler in asphalt mastic, with the aim of increasing the use of sustainable road construction materials. In this study, X-ray fluorescence spectroscopy (XRF) and scanning electron microscopy (SEM) were used to characterize the oxide composition and micromorphology of limestone (LS), red mud (RM), steel slag (SS), and ground granulated blast-furnace slag (GGBFS). Four asphalt mastics containing LS, RM, SS, and GGBFS with a filler-to-binder weight ratio of one were prepared. An evaluation of the rheology and wetting of the solid-waste-filler asphalt mastic was conducted using a frequency sweep, temperature sweep, linear amplitude sweep (LAS), multiple stress creep and recovery (MSCR), and surface free energy (SFE) methods. The results showed that SS increased the complex modulus, elastic component of the asphalt mastic and decreased the nonrecoverable creep compliance at stress levels of 0.1 and 3.2 kPa, which improved the rutting resistance of the asphalt mastic and reduced deformation under high-temperature conditions. The RM and GGBFS increased the fatigue performance of the asphalt mastic under strain loading, enhanced its fatigue life, and maintained good performance under long-term loading. The dispersive component of the SFE parameter of the solid-waste-filler asphalt mastic was larger than the polar component for the largest share of the surface energy composition. The SFE of the asphalt mastic prepared from the industrial solid-waste filler was reduced; however, the difference was insignificant compared to the limestone asphalt mastic. Solid-waste-filler asphalt mastic has performance characteristics, and its actual application can be based on different performance characteristics to select an appropriate solid-waste filler. The results of this study provide new technological solutions for solving the utilization rate of solid waste materials and sustainable road construction in the future.