Traffic light detection and recognition are crucial for enhancing the security of unmanned systems. This study proposes a YOLOv5-based traffic light-detection algorithm to tackle the challenges posed by small targets and complex urban backgrounds. Initially, the Mosaic-9 method is employed to enhance the training dataset, thereby boosting the network’s ability to generalize and adapt to real-world scenarios. Furthermore, the Squeeze-and-Excitation (SE) attention mechanism is incorporated to improve the network. Moreover, the YOLOv5 algorithm’s loss function is optimized by substituting it with Efficient Intersection over Union loss (EIoU_loss), which addresses issues like missed detection and false alarms. Experimental results demonstrate that the model trained with this enhanced network achieves an mAP (mean average precision) of 99.4% on a custom dataset, which is 6.3% higher than that of the original YOLOv5, while maintaining a detection speed of 74 f/s. Therefore, this algorithm offers higher detection accuracy and effectively meets real-time operational requirements. The proposed method has a strong application potential, and can be widely used in the field of automatic driving, assisted driving, etc. Its application is not only of great significance in improving the accuracy and speed of traffic sign detection, but also can provide technical support for the development of intelligent transportation systems.