The traditional traffic sign detection algorithm can not deal with the application scenarios such as intelligent transportation system or advanced assisted driving environment, and it is difficult to meet the application requirements in detection accuracy and efficiency. Focusing on the above problems, this paper proposes a traffic sign detection algorithm based on Single Shot Multibox Detector (SSD) combined with Receptive Field Module (RFM) and Path Aggregation Network (PAN). The proposed algorithm is abbreviated to SSD-RP. The SSD-RP uses the RFM to improve the receptive field and semantics of the predicted feature maps, thus improving the detection performance of small traffic signs. At the same time, the path aggregation network is introduced to integrate multiscale features, which makes the abstract semantic information and rich detailed information shared among multiscale feature maps, enhances the discrimination ability of feature system, and improves the location and classification accuracy of traffic signs. Following that, the spatial pyramid pooling module is used to pool the shallow features and integrate them into the bottom-up information transmission path of the path aggregation network, thus continuing to supplement the fine-grained features for the feature system and further improve the detection performance. The experimental results on GTSDB and CCTSDB data sets show that SSD-RP has higher mean average precision (mAP) than traditional SSD algorithm and can better detect small traffic signs, which means that SSD-RP has higher detection precision. In addition, the experimental results also show that, compared with the common object detection algorithms such as Faster R-CNN, RetinaNet, and YOLOv3, the SSD-RP achieves a better balance between detection time and detection precision.