In this study, the effect of seven industrial waste materials as mineral fillers in asphalt mixtures was investigated. Silica fume (SF), limestone dust (LSD), stone dust (SD), rice husk ash (RHA), fly ash (FA), brick dust (BD), and marble dust (MD) were used to prepare the asphalt mixtures. The obtained experimental results were compared with ordinary Portland cement (OPC), which is used as a conventional mineral filler. The physical, chemical, and morphological assessment of the fillers was performed to evaluate the suitability of industrial waste to replace the OPC. The volumetric, strength, and durability of the modified asphalt mixes were examined to evaluate their performance. The experimental data have been processed through artificial neural networks (ANNs), using k-fold cross-validation as a resampling method and two different activation functions to develop predictive models of the main mechanical and volumetric parameters. In the current research, the two most relevant parameters investigated are the filler type and the filler content, given that they both greatly affect the asphalt concrete mechanical performance. The asphalt mixes have been optimized by means of the Marshall stability analysis, and after that, for each different filler, the optimum asphalt mixtures were investigated by carrying out Indirect tensile strength, moisture susceptibility, and abrasion loss tests. The moisture sensitivity of the modified asphalt mixtures is within the acceptable limit according to the Indian standard. Asphalt mixes modified with the finest mineral fillers exhibited superior stiffness and cracking resistance. Experimental results show higher moisture resistance in calcium-dominant mineral filler-modified asphalt mixtures. Except for mixes prepared with RHA and MD (4% filler content), all the asphalt mixtures considered in this study show MS values higher than 10 kN, as prescribed by Indian regulations. All the values of the void ratio for each asphalt mix have been observed to range between 3–5%, and MQ results were observed between 2 kN/mm–6 kN/mm, which falls within the acceptable range of the Indian specification. Partly due to implementing a data-augmentation strategy based on interpolation, the ANN modeling was very successful, showing a coefficient of correlation averaged over all output variables equal to 0.9967.