Mineral fillers are added into asphalt mixtures to stiffen asphalt binder and improve mixture density and strength. The stiffness of the mastic affects the ability of the mixture to resist permanent deformation at higher temperatures, influences stress development and fatigue resistance at intermediate temperatures, and influences stress development and fracture resistance at low temperatures. In this study, the effect of mineral fillers and their fractional voids on mastic rheological and mechanical properties is evaluated. Four types of filler with different physical properties were investigated. The fillers were associated to four asphalt binders, including unmodified and polymer modified binders, to obtain 16 asphalt mastics. The rheological properties of the mastics were evaluated conducting Superpave testing procedures, including the dynamic shear rheometer (DSR) and the bending beam rheometer (BBR). Their mechanical response was investigated using a Modified Direct Tension Test (MDTT), also employing a completely redesigned in-house developed Digital Image Correlation System (DIC) capable of accurately capturing localized or non-uniform stress distributions. The results showed that the Rigden fractional void is an effective indicator of filler stiffening effect, but it cannot be employed to evaluate the interaction between asphalt binder and filler. The effect of filler volume fraction on mastic performance depends on the binder employed.