To explore the interactions and mechanisms of Thermoplastic polyurethane (TPU)-modified asphalt with different kinds of mineral fillers, a micro-mechanical model for TPU-modified asphalt mastic was established, which considered the interaction between asphalt and mineral powder to effectively analyze the internal mechanisms affecting the rheological properties of TPU-modified asphalt mastic. In this study, according to the micro-mechanics of composites’ principles, the dynamic shear modulus (|G*|) of asphalt mastic with different mass ratios of filler/asphalt (F/A) was calculated by the homogenize morphologically representative pattern (H-MRP) model. The key ratio of F/A, which is close to the test result, can be determined, and a four-phase H-MRP model of the TPU modified asphalt mastic was established after considering the structure of asphalt layer thickness. The results were interpreted based on the known reactions of TPU with asphalt model compounds. The |G*| of TPU-modified asphalt mastic was predicted by using this model. Furthermore, the effects of the complex shear modulus, Poisson’s ratio of TPU-modified asphalt, Poisson’s ratio, particle size of mineral powder, and thickness of the structural asphalt layer in the |G*| of TPU-modified asphalt mastic were analyzed in the whole-model construction, as well as the internal mechanism of the |G*| of TPU modified asphalt mastic. In addition, can also be found the predicted value of |G*| calculated by the four-phase H-MRP model is close to the experimental value after choosing a structural asphalt layer of appropriate thickness.