The decomposition number K and penalty factor α in the variational mode decomposition (VMD) algorithm have a great influence on the decomposition effect and the accuracy of subsequent fault diagnosis. Therefore, a gear fault diagnosis method based on genetic mutation particle swarm optimization VMD and probabilistic neural network (GMPSO-VMD-PNN) algorithm is proposed in this paper. Firstly, the GMPSO algorithm is used to optimize the [K , α] parameter combination in the VMD algorithm, and the optimal [K , α] parameter combination of each gear fault vibration signal to be decomposed is selected. Then, the gear fault vibration signal is decomposed into several intrinsic mode functions (IMFs) by VMD, and the sample entropy value of each IMFs is extracted to form the feature vector of subsequent fault diagnosis. Finally, the characteristic vector of gear fault vibration signal is input into PNN model, and gear fault is accurately classified. By comparing with fixed parameter VMD algorithm, empirical mode decomposition (EMD) and complete ensemble empirical mode decomposition adaptive noise (CEEMDAN) algorithm, the superiority of this method in gear fault diagnosis is verified. Therefore, the GMPSO-VMD-PNN algorithm proposed in this paper has certain application value for gear fault diagnosis.