Recently, China has started paying more attention to environmental protection, and the efficient utilization of exhaust gases produced by smelting has emerged as a key problem concern. e silica fume collected from the exhaust gases produced by smelting ferrosilicon or industrial silicon was often used as a cement concrete admixture. Using silica fume as an asphalt modifier can make exhaust gases profitable. In this study, silica fume/SBS composite-modified asphalt mortar was prepared to improve the performance of asphalt. e effects of the silica fume content, temperature, and ratio of filler asphalt on the composite-modified asphalt mortar were studied through the cone penetration, softening point, viscosity, dynamic shear rheological (DSR) test, and bending beam rheometer (BBR) test. e thermal stability of composite-modified asphalt was analyzed through the thermal analysis test. e results showed that with the increase of silica fume content and ratio of filler asphalt, the high-temperature performance of asphalt mortar was improved; the content of silica fume had a great influence on the low-temperature performance of asphalt mortar, the optimum silica fume content was 7%; increasing the ratio of filler asphalt reduced the low-temperature cracking resistance of asphalt mortar; the incorporation of silica fume enhanced the initial decomposition temperature, thermal residual rate, temperature in which the weight loss rate reaches the maximum, and the endothermic peak of maximum temperature and improved the properties of asphalt.