Pedestrians have random distribution and dynamic characteristics. Aiming to this problem, this paper proposes a pedestrian object detection method based on improved YOLOv5 in urban road scenes. Firstly, the last C3 module was replaced in the Backbone with the SE attention mechanism to enhance the network’s extraction of pedestrian object features and improve the detection accuracy of small-scale pedestrians. Secondly, the EIOU loss function was introduced to optimize the object detection performance of the detection network. To validate the effectiveness of the algorithm, experiments were conducted on a dataset composed of filtered Caltech pedestrian detection data and images taken by ourselves. The experiments showed that the improved algorithm has P-value, R-value, and mAP of 98.4%, 95.5%, and 98%, respectively. Compared to the YOLOv5 model, it has increased P-value by 1.4%, R-value by 2.7%, and mAP by 1.3%. The improved algorithm also boosts the detection speed. The detection speed is 0.8 ms faster than the YOLOv5 model. It is also faster than other mainstream algorithms including Faster R-CNN and SSD. The improved algorithm enhances the effectiveness of pedestrian detection significantly and has important application value.