Two of the main problems encountered in flexible pavements are the stripping of coarse aggregates and the formation of rut depth due to increases in the volume of road traffic and heavy vehicle loads, especially in areas where speeds are low. The existence of rut depth also affects the comfort and safety of road users due to the water accumulation on the pavement surface and reducing tire/pavement friction, which can lead to hydroplaning phenomena. In this research, it was proven that the use of fillers of different origins influences the affinity between aggregates and the binder. The effect of an adhesion promoter in the mix design (such as the amine included in cellulosic fiber pellets) was also studied. Several tests were carried out to determine the binder/aggregate adhesiveness, water sensitivity and resistance to permanent deformation, to evaluate the performance of different blends. It was found that the addition of this additive increased 10% of the aggregate surfaces covered with bitumen when compared with the aggregates without this addition. As expected, the water sensitivity tests showed that the mixture with granitic filler had the lowest indirect tensile strength ratio (ITSR) value (70%), while the mixtures with limestone filler led to the highest percentages (ranging from 83 to 93%). As for the results of the wheel tracking tests (WTT), it was confirmed that the use of limestone filler translates into an improvement in the performance against the permanent deformation of the asphalt mixtures. The mixture with higher bitumen content and adhesion promoter revealed the best average results.