In real scenarios, accurate and real‐time detection of traffic signs is of great significance to the automatic driving system. To meet the requirements of detection accuracy and speed, a new traffic sign detection method based on YOLOv5 and Swin‐Transformer is proposed in this paper. First, based on the traditional Focus structure, a lightweight shallow information enhancement module is designed. Second, to enhance the channel weights of useful information, an adjustable channel attention mechanism is proposed. Additionally, a Cross Stage Partial module based on Swin‐Transformer is designed to capture contextual information around traffic signs, thereby improving the detection accuracy of small‐scale traffic signs. Finally, to better fuse deep semantic features and shallow detail features, an adaptive feature fusion method is proposed. To verify the superiority of the proposed method, experimental verification was carried out on TT100K and DFG traffic sign detection datasets, and their mAP, AP50 and FPS were (75.3, 94.8, 82) and (79.7, 85.9, 118), respectively. The experimental results show that the proposed traffic sign detection method has high accuracy and real‐time performance, and can meet the needs of traffic sign detection in the actual scene.