Variational mode decomposition (VMD) is widely used in the condition monitoring and fault diagnosis of rotary machinery for its unique advantages. An adaptive parameter optimized VMD (APOVMD) is proposed in order to adaptively determine the suitable decomposed parameters and further enhance its performance. The traditional singular value decomposition (SVD) method cannot effectively select the reconstructed order, which often leads to unsatisfactory results for signal reconstruction. Thus, a singular kurtosis difference spectrum method is proposed to accurately determine the effective reconstructed order for signal noise reduction. In addition, because the fault signal of the planetary gearbox at the early fault stage is weak and susceptible to ambient noise and other signal interference, the fault feature information is difficult to extract. To address this issue, a novel method for early fault feature extraction of planetary gearbox based on APOVMD and singular kurtosis difference spectrum is proposed in this paper. First, the APOVMD is applied to decompose the planetary gearbox vibration signal into a series of band-limited intrinsic mode functions adaptively and non-recursively. Second, the sensitive component is selected from the IMFS according to the cosine similarity index. Third, the Hankel matrix is constructed for the sensitive component in the phase space and decomposed by SVD. Here, the effective reconstructed order is automatically selected by the singular kurtosis difference spectrum method for noise reduction. Finally, the Hilbert envelope spectrum analysis is carried out on the reconstructed signal, and the fault characteristic frequency information of planetary gearbox can be accurately extracted from the envelope spectrum to realize the fault identification and location. The results of simulation studies and actual experimental data analysis demonstrate that the proposed method has superior ability to extract the early weak fault characteristics of the planetary gearbox compared with the VMD-SVD and EEMD-SVD methods, and the validity and feasibility of the presented method are proved. INDEX TERMS Planetary gearbox, adaptive parameter optimized VMD, singular kurtosis difference spectrum, cosine similarity, early fault diagnosis.