Aiming at the problem that mining conveyor belts are easily damaged under severe working conditions, the paper proposed a deep learning-based conveyor belt damage detection method. To further explore the possibility of the application of lightweight CNNs in the detection of conveyor belt damage, the paper deeply integrates the MobileNet and Yolov4 network to achieve the lightweight of Yolov4, and performs a test on the exiting conveyor belt damage dataset containing 3000 images. The test results show that the lightweight network can effectively detect the damage of the conveyor belt, with the fastest test speed 70.26 FPS, and the highest test accuracy 93.22%. Compared with the original Yolov4, the accuracy increased by 3.5% with the speed increased by 188%. By comparing other existing detection methods, the strong generalization ability of the model is verified, which provides technical support and empirical reference for the visual monitoring and intelligent development of belt conveyors.