To address the inaccurate classification of coarse aggregate particle size due to insufficient height information in single‐view, a multi‐view and graph convolutional network (GCN) based method for coarse aggregate particle size classification was proposed in this study. First, the viewpoint selection and projection strategies were designed to build the aggregate multi‐view datasets. Then, the surface texture of the aggregate was reconstructed by using 3D point cloud information. Finally, self‐attention mechanism and three‐layer GCN were introduced to aggregate global shape feature descriptors. The experimental results show that the proposed interleaved self‐attention and view GCN model achieves a coarse aggregate particle size classification accuracy of 94.11%, outperforming other multi‐view classification algorithms. This method provides a new possibility for the accurate detection of aggregate particle size and provides significant support for the production and automatic detection of aggregate raw materials.