With the increasing number of electric vehicles, V2G (vehicle to grid) charging piles which can realize the two-way flow of vehicle and electricity have been put into the market on a large scale, and the fault maintenance of charging piles has gradually become a problem. Aiming at the problems that convolutional neural networks (CNN) are easy to overfit and the low localization accuracy in fault diagnosis of V2G charging piles, an improved fault classification model based on convolutional neural networks (CNN-SVM) is proposed. Firstly, the hardware adaptation optimization is carried out for the CNN structure, the wavelet packet transformation is used to extract the fault current signal feature information into the CNN, and the CNN-SVM model is constructed by SVM (Support Vector Machine) instead of the SoftMax classifier in the CNN. The PSO (particle swarm algorithm) is used to optimize the parameters of the SVM model to obtain the optimal model. Finally, the superiority of the proposed method is verified by multi-working cases. The experimental results show that the fault classification accuracy of the CNN-SVM model is far higher than that of the traditional deep learning network and has practical significance for fault diagnosis of the switch module of the charging pile.