To obtain the viscoelastic parameters of asphalt mixtures and analyze the effect of temperatures and modifiers on viscoelastic properties of asphalt mixtures, the creep compliances of the neat asphalt mixture (AM), compound diatomite and basalt fibers reinforced asphalt mixture (DBFAM), and styrene-butadiene-styrene modified asphalt mixture (SBSAM) were tested and calculated by the static creep tests. And the creep compliances of the three asphalt mixtures at −20 °C, −10 °C, and 0 °C are deducted by the time–temperature equivalence principle (TTEP) and Arrhenius equation. Further, the relaxation modulus of the three asphalt mixtures from −20 °C to 50 °C at 10 °C increments are calculated by the convolution integral and Simpson method. Subsequently, the Burgers model, the generalized Kelvin model, and the generalized Maxwell model are applied to analyze the viscoelastic properties of the three asphalt mixtures at different temperatures. The results show that the generalized Kelvin model and the generalized Maxwell model are superior to the Burgers model in describing the variation of viscoelastic properties of asphalt mixtures with loading time. At low temperatures, asphalt mixtures have excellent properties in resisting permanent deformation and releasing internal stress. Besides, the addition of SBS modifier and compound diatomite and basalt fibers modifier can significantly raise the viscosity η1 and the elastic modulus E1 of the asphalt mixture, respectively.