In order to improve the accuracy and universality of the nonlinear viscoelastic-plastic mechanical behavior characterization method of asphalt mixture, a new criterion for the division of the creep process of materials was established based on the strain yield characteristics, and the coexistence mechanism of Viscoelastic–Viscoplastic strain was proposed in the subsequent yield phase; then, a viscoelastic element was constructed in the form of a parallel connection of two fractional viscoelastic elements based on fractional calculus theory, and its mathematical equations were derived; with novel viscoelastic elements, a constitutive model characterizing the whole creep process of asphalt mixtures was developed and its analytical expression was derived. The laboratory short-term creep test of Cement and Asphalt Mortar (CA mortar) and the simulation test data of asphalt mixtures from the references were used to verify the constitutive model. The results show that the creep constitutive model of asphalt mixture established in this paper has excellent fitting accuracy for different phases of the creep process of asphalt mixture under different stress levels, where the minimum fitting correlation values R2 for CA mortar, asphalt mixture (applied to pavement engineering), and asphalt sand are 0.9976, 0.981, and 0.979, respectively. Therefore, this model can be used to provide a theoretical reference for the study of the characterization of the mechanical behavior of asphalt materials.