It is very important for pavement engineers to know which factors are the main reasons for the damage of the paving around manholes. Based on the investigation on the damage of paving around manholes, a vibration model with multidegree of freedom for the vehicle-manhole cover was established and analyzed. After that, the Matlab software was used to obtain the variation law of impact load over time, and the 95% fourth power of the aggregate force was used as the index to evaluate the pavement damage. Finally, many influencing factors on pavement damage were analyzed by the method of grey correlation entropy. The results indicated that the impact load reached the maximum for the first time when the vehicle reached the top of the manhole cover, which was 1.29 times that of the static load, and the pavement damage coefficient was 2.12 times that of the static load. The influencing factors had different degrees of influence on the pavement damage; from large to small, they were change of road longitudinal slope > driving speed > damping of tire > stiffness of tire > height difference from the pavement damage > height difference from the manhole settlement > stiffness of the manhole cover.