In the daily inspection task of the expressway, accuracy and speed are the two most important indexes to reflect the detection efficiency of nondeformation diseases of asphalt pavement. To achieve model compression, accelerated detection, and accurate identification under multiscale conditions, a lightweight algorithm (PDNet) based on improved YOLOv5 is proposed. The algorithm is improved based on the network structure of YOLOv5, and the improved network structure is called YOLO-W. Firstly, a novel cross-layer weighted cascade aggregation network (W-PAN) is proposed to replace the original YOLOv5 network. Secondly, more economical GhostC3 and ShuffleConv modules are designed to replace C3 and Conv modules in the original network model. In terms of parameter setting, CIoU is selected as the loss function of the model, and the K-Means ++ algorithm is used for anchor box clustering. Before the model training, the confrontation generation network (GAN) and Poisson migration fusion algorithm (Poisson) are used for data enhancement and the negative sample training (NST) method is used to improve the robustness of the model. Finally, Softer-NMS is used to remove the prediction box in the prediction stage. Seven common asphalt pavement disease data sets (FAFU-PD) are constructed at the same time. Compared with the original YOLOv5 algorithm, PDNet improves the scores of FAFU-PD data sets on F1-score by 10 percentage points and FPS by 77.5%.