Aggregate occupies at least three-quarters of the volume of asphalt mixture and can significantly affect the performance of pavement. The geometrical morphology influences the slippage and interlock among aggregates for resisting and distributing applied loads. In recent years, the discrete element method (DEM) has been employed for simulation of asphalt mixture structure. This paper introduces an approach for simulation of aggregate and asphalt mixture using parameterized shape and size gradation. Both plane geometry factor (PGF) and section aspect ratio (SAR) were employed to describe the 3D geometric characteristics of aggregates. A numerical technique of aggregate models was implemented with probabilistic parameters depending on statistical results of PGFs and SARs. Therefore, the 3D numerical model of asphalt mixtures was assembled with three different components, which is validated by uniaxial compression test via comparison with that of the laboratory result. It was found that the PGF and SAR are appropriate to describe the three-dimensional features of aggregate shapes, due to the fact that a simplified space object can be described by a 2D graphical projection and a vector scalar corresponding to the space vector. Probability distribution curves of PGFs and SARs between coarse aggregates are in concordance with the Gauss-type function, since their correlation coefficients are all greater than 95%. It was verified that the developed clumping algorithm of aggregates was reasonable with the shapes and size gradation. Based on the parallel-bond model and the Burger's model, the results of virtual tests are in good agreement with those of laboratory uniaxial tests. It is shown that the angularity (PGF) of aggregates has a beneficial effect on the strength and stability while the flat-elongated feature (SAR) has a negative effect on those of asphalt mixtures.