The machine vision system was employed to inspect the surface defects of bridge cables of cable-stayed bridges. After the acquisition and preprocessing of the defect images, it is necessary to classify and identify the defects of the cables to meet the requirements of non-destructive testing and evaluation. In this paper, feature extraction for defect images was performed using mathematical statistical methods. After that, 10 feature parameters including shape features, grayscale features and texture features of the defect images were obtained and selected for a classification model of support vector machine (SVM). To improve the SVM classification performance, the particle swarm optimization algorithm (PSO) was adopted to obtain the punish factor c and the kernel parameter g of the SVM model, namely the PSO-SVM algorithm. Finally, our PSO-SVM classification model was employed to implement the classification of real surface defect images of the bridge cables. Longitudinal crack, transverse crack, surface corrosion, and pothole defect can be automatically identified and the classification accuracy reached 96.25%. The experimental results showed that the PSO-SVM model can improve the classification performance of the surface defects. Based on the effective classification, we can find the distribution characteristics of the surface defects of the cable. It is very important to analyze the relationship between the type of surface defects and the material of the protective layer, so as to adopt appropriate materials and reasonable maintenance measures. Thus, it has a great significance in the structural health monitoring of bridge cables.