In this study, a depth camera-based intelligence method is proposed. First, road damage images are collected and transformed into a training set. Then training, defect detection, defect extraction, and classification are performed. In addition, a YOLOv5 is used to create, train, validate, and test the label database. The method does not require a predetermined distance between the measurement target and the sensor; can be applied to moving scenes; and is important for the detection, classification, and quantification of pavement diseases. The results show that the sensor can achieve plane fitting at investigated working distances by means of a deep learning network. In addition, two pavement examples show that the detection method can save a lot of manpower and improve the detection efficiency with certain accuracy.