Wearing safety helmets can effectively protect workers safety on construction sites. However, workers often take off the helmets because of weak security-conscious and discomfort, then hidden dangers will be brought by this behaviour. Workers without safety helmets will suffer more injuries in accidents such as falling human body and vertical falling matter. Hence, detecting safety helmet wearing is a vital step of construction sites safety management and a safety helmet detector with high speed and accuracy is urgently needed. However, traditional manual monitor is labour intensive and methods of installing sensors on safety helmet are difficult to popularize. Therefore, this paper proposes a deep learning-based method to detect safety helmet wearing at a satisfactory accuracy with high detection speed. Our method chooses YOLO v5 as the baseline, then the fourth detection scale is added to predict more bounding boxes for small objects and the attention mechanism is adopted in the backbone of the network to construct more informative features for following concatenation operations. In order to overcome the defects caused by insufficient data, targeted data augmentation and transfer learning are used. Improvements caused by every modification are discussed in this paper. Finally, our model achieves 92.2% mean average precision, up 6.3% compared to the original algorithm, and it only takes 3.0 ms to detect an image at 640 × 640. These results demonstrate the robustness and feasibility of our model. Meanwhile, the size of our trained model is only 16.3 m, which means the model is easy to be deployed. At last, after obtaining a satisfactory model, a graphical user interface (GUI) is designed to make our algorithm more user-friendly.INDEX TERMS Safety helmet wearing detection, YOLO v5, four detection scales, attention mechanism, GUI design.