Instantaneous frequency estimation of rolling bearing is a key step in order tracking without tachometers, and time-frequency analysis method is an effective solution. In this paper, a new method applying the variational mode decomposition (VMD) in association with the synchroextracting transform (SET), named VMD-SET, is proposed as an improved time-frequency analysis method for instantaneous frequency estimation of rolling bearing. The SET is a new time-frequency analysis method which belongs to a postprocessing procedure of the short-time Fourier transform (STFT) and has excellent performance in energy concentration. Considering nonstationary broadband fault vibration signals of rolling bearing under variable speed conditions, the time-frequency characteristics cannot be obtained accurately by SET alone. Thus, VMD-SET method is proposed. Firstly, the signal is decomposed into several intrinsic mode functions (IMFs) with different center frequency by VMD. Then, effective IMFs are selected by mutual information and kurtosis criteria and are reconstructed. Next, the SET method is applied to the reconstructed signal to generate the time-frequency representation with high resolution. Finally, instantaneous frequency trajectory can be accurately extracted by peak search from the time-frequency representation. The proposed method is free from time-varying sidebands and is robust to noise interference. It is proved by numerical simulated signal analysis and is further validated by lab experimental rolling bearing vibration signal analysis. The results show this method can estimate the instantaneous frequency with high precision without noise interference.