Porous asphalt (PA) pavements are widely employed in areas with wet climates due to their excellent permeability and superior performance. As particle enhancement inclusions in asphalt mastic, mineral fillers play essential roles in improving the performance of PA pavements. This study developed a coupled multiscale finite element (FE) model, involving the mesostructure of PA mixture and PA pavement. Within this model, the mesoscale structure was captured by X-ray computer tomography (X-ray CT) scanning and reconstructed with digital image processing (DIP) technology. Four types of mastic properties were employed with four mineral fillers (Granodiorite, Limestone, Dolomite, and Rhyolite) in the mesoscale portion of the pavement model to analyze the effects of filler types on the performance of pavements. A constant tire loading was applied and two temperatures (0 °C and 50 °C) were specified. The performances (load-bearing capacity, rutting resistance, and raveling resistance) of pavements with different fillers were identified and ranked, and their correlations with the chemical components of the four fillers were analyzed. The computational results showed that pavements with Rhyolite and Granodiorite fillers have higher load-bearing capacities and rutting resistance, while the Limestone and Dolomite fillers can improve the raveling resistance of the PA pavements. In the correlation analysis, the chemical components Al2O3 and SiO2 play dominant roles in improving the load-bearing capacities and rutting resistance of the PA pavements, and the fillers with high percentages of CaO can improve the raveling resistance of the PA pavements. Based on this algorithm, it is possible to select an optimal filler for a specific pavement design and thus improve the durability of the PA pavements.