Morphological characteristics of aggregates have direct impacts on performances of asphalt mixes. This paper aims to investigate the effects of the morphological characteristics of fine and coarse aggregates on the high-temperature viscoelastic properties of asphalt mortars and mixtures. For this purpose, an experimental proportion scheme was designed for asphalt mixes prepared with three different types of aggregates (basalt, andesite and pebble/river sand) based on the simplex lattice design (SLD) method. Three morphological parameters were chosen to characterize shape, angularity and texture of aggregates. Afterwards, the uniaxial compression creep test was conducted for asphalt mixes and the high-temperature viscoelastic properties were obtained based on Burgers model. The effects of fine and coarse aggregates on the viscoelastic properties are analyzed through asphalt mortars and mixtures, respectively. The results showed that aggregate morphological characteristics correlate with the high-temperature viscoelastic properties of asphalt mixes, especially for fine aggregates. Aggregates with complex morphological characteristics are conducive to improving the deformation recovery and anti-deformation of asphalt mixes. Furthermore, coarse aggregates can enhance the anti-deformation of asphalt mixture effectively due to its skeleton effect.