In this work, to quantitatively analyze the roughness of the surfaces of road aggregates, the contact measurement technique and contactless scanning technique were, respectively, used to capture the coordinate data of point clouds on the aggregate surface, which were then used to reconstruct the digital elevation models of aggregate particles. Then, the joint roughness coefficient (JRC) was used as an evaluation index, and the quantitative calculation methods of the two-dimensional (2D) contour line roughness and three-dimensional (3D) contour surface roughness of aggregate particles were, respectively, studied. Finally, the anisotropic characteristics and size effect of the roughness coefficients of aggregates with different lithologies were, respectively, investigated, based on which the practicability of the 3D roughness coefficient index was proven. The results demonstrate that the roughness of a road aggregate surface can be quantitatively described by the point cloud data. The 2D roughness of aggregate profile lines exhibits anisotropy, while the 3D roughness of the aggregate contour surface indicates the size effect. The subtle morphological changes of the surface textures of aggregates can be accurately described by the 3D joint roughness coefficient (JRC3D) calculated by the feature parameter method.