The feature extraction problem of coupled vibration signals with multiple fault modes of planetary gears has not been solved effectively. At present, kernel principal component analysis (KPCA) is usually used to solve nonlinear feature extraction problems, but the kernel function selection and its blind parameter setting greatly affect the performance of the algorithm. For the optimization of the kernel parameters, it is very urgent to study the theoretical modeling to improve the performance of kernel principal component analysis. Aiming at the deficiency of kernel principal component analysis using the single-kernel function for the nonlinear mapping of feature extraction, a dual-kernel function based on the flexible linear combination of a radial basis kernel function and polynomial kernel function is proposed. In order to increase the scientificity of setting the kernel parameters and the flexible weight coefficient, a mathematical model for dual-kernel parameter optimization was constructed based on a Fisher criterion discriminant analysis. In addition, this paper puts forward a swarm intelligent fusion algorithm to increase this method’s advantages for optimization problems, involving the shuffled frog leaping algorithm combined with particle swarm optimization (SFLA-PSO). The new fusion algorithm was applied to optimize the kernel parameters to improve the performance of KPCA nonlinear mapping. The optimized dual-kernel function KPCA (DKKPCA) was applied to the feature extraction of planetary gear wear damage, and had a good identification effect on the fuzzy damage boundary of the planetary gearbox. The conclusion is that the DKKPCA optimized by the SFLA-PSO swarm intelligent fusion algorithm not only effectively improves the performance of feature extraction, but also enables the adaptive selection of parameters for the dual-kernel function and the adjustment of weights for the basic kernel function through a certain degree of optimization; so, this method has great potential for practical use.