Pavement disease detection and classification is one of the key problems in computer vision and intelligent analysis. This is an automated target detection technology with great development potential, which can improve the detection efficiency of road management departments. The research based on the convolutional neural network is aimed at realizing asphalt pavement disease detection based on low resolution, occlusive interference, and complex environment. Considering the powerful function of the convolutional neural network and its successful application in object detection, we apply it to asphalt pavement disease detection, and the detection results are used for subsequent analysis and decision-making. At present, most of the research on pavement disease detection focuses on crack detection, and the detection of multiclass diseases is less, and its detection accuracy and speed need to be improved, which does not meet the actual engineering application. Therefore, a rapid asphalt pavement disease detection method based on improved YOLOv5s was proposed. The complex scene data enhancement technique was developed, which is used to enhance and extend the original data to improve the robustness of the model. The improved lightweight attention module SCBAM was integrated into the backbone network, which can enhance the feature extraction ability and improve the detection performance of the model for small targets. The spatial pyramid pooling was improved into SPPF to fuse the input features, which can solve the multiscale problem of the target and improve the reasoning efficiency of the model to a certain extent. The experimental results showed that, after the model is improved, the average accuracy of pavement disease reaches 94.0%. Compared with YOLOv5s, the precision of the improved YOLOv5-pavement is increased by 3.1%, the recall rate is increased by 4.4%, the F1 score is increased by 3.7%, and the mAP is increased by 3.8%. For transverse cracks, longitudinal cracks, mesh cracks, potholes, and repaired pavement, the detection accuracy of pavement disease detection method based on YOLOv5-pavement is improved by 3.4%, 3.1%, 4.0%, 7.5%, and 4.8%, respectively, compared with that based on YOLOv5s. The proposed method provides support for the detection work of pavement diseases.