Variational mode decomposition (VMD) with a non-recursive and narrow-band filtering nature is a promising time-frequency analysis tool, which can deal effectively with a non-stationary and complicated compound signal. Nevertheless, the factitious parameter setting in VMD is closely related to its decomposability. Moreover, VMD has a certain endpoint effect phenomenon. Hence, to overcome these drawbacks, this paper presents a novel time-frequency analysis algorithm termed as improved adaptive variational mode decomposition (IAVMD) for rotor fault diagnosis. First, a waveform matching extension is employed to preprocess the left and right boundaries of the raw compound signal instead of mirroring the extreme extension. Then, a grey wolf optimization (GWO) algorithm is employed to determine the inside parameters ( α ^ , K) of VMD, where the minimization of the mean of weighted sparseness kurtosis (WSK) is regarded as the optimized target. Meanwhile, VMD with the optimized parameters is used to decompose the preprocessed signal into several mono-component signals. Finally, a Teager energy operator (TEO) with a favorable demodulation performance is conducted to efficiently estimate the instantaneous characteristics of each mono-component signal, which is aimed at obtaining the ultimate time-frequency representation (TFR). The efficacy of the presented approach is verified by applying the simulated data and experimental rotor vibration data. The results indicate that our approach can provide a precise diagnosis result, and it exhibits the patterns of time-varying frequency more explicitly than some existing congeneric methods do (e.g., local mean decomposition (LMD), empirical mode decomposition (EMD) and wavelet transform (WT) ).