Traffic signal recognition is a critical function of the intelligent vehicle system (IVS). Many algorithms can achieve a high accuracy in traffic signal recognition. But these algorithms have poor generalization ability, and their recognition rates vary greatly with datasets. These defects hinder their application in unmanned driving. To solve the problem, this paper introduces self-paced learning (SPL) to the image recognition of traffic signs. Based on complexity, the SPL automatically classifies samples into multiple sets. If machine learning (ML) algorithm is trained by the sample sets in ascending order of complexity, a universal computing model will be obtained, and the ML algorithm will have a better generalization ability. Here, the support vector machine (SVM) is adopted as the classifier for traffic sign detection, and the convolutional neural network (CNN) is employed as the classifier for traffic sign recognition. Then, the two classifiers were trained by the SPL on two public datasets: German Traffic Sign Detection Benchmark (GTSDB) and German Traffic Sign Recognition Benchmark (GTSRB). The model obtained through the training was tested on Belgium Traffic Sign Detection Benchmark (BTSDB) and KITTI datasets. The results show that the obtained computing model achieved similar accuracy on the training sets and test sets. Hence, the SPL can indeed enhance the generalization ability of ML algorithms, and promote the application of CNN, SVM, and other ML algorithms in unmanned driving.