This research aims to improve the durability of skid resistance of asphalt pavement from the perspective of coarse aggregates based on on-site investigation. Firstly, the skid resistance of six representative actual roads was tested during two years by employing the Dynamic Friction Tester and the attenuation characteristics of skid resistance of different types of asphalt pavements were analyzed. Secondly, core samples were drilled onsite and coarse aggregates were extracted from the surface layer of the core samples. The morphological parameters of coarse aggregates were collected by a “backlighting photography” system and three-dimensional profilometer, and the variation rules of angularity and micro-texture of coarse aggregates were investigated. Finally, the correlation between the morphological characteristics of coarse aggregates and the pavement skid resistance was established based on the grey correlation entropy. The research results show that with the increase in service time, the attenuation rate of skid resistance of asphalt pavement gradually slows down; the angularity of coarse aggregates gradually decreases, and the micro-texture on the wearing surface gradually wears away. The grey correlation entropy between all the micro-texture indexes of coarse aggregates and dynamic friction coefficient, as well as between the roundness and skid resistance is more than 0.7, whereas the correlation between other evaluation indicators and the dynamic friction coefficient is poor, indicating that compared with the angularity of coarse aggregates, the micro-texture affects the skid resistance of actual asphalt pavement more greatly. In engineering applications, the use of coarse gradation, coarse aggregates with high roughness or high anti-wear performance can slow down the attenuation of pavement skid resistance, so that the pavement can maintain superior long-term anti-skidding performance.