The time-frequency energy distribution processed by a short-time Fourier transform can be compressed to the real instantaneous frequency by the synchrosqueezing transformation (SST), which improves the time-frequency energy concentration of the signal. However, there is a large error in the instantaneous frequency estimation of a multicomponent nonpure harmonic signal by the SST. Therefore, a method for determining the instantaneous frequency (IF) of a rolling bearing based on a fractional synchrosqueezing transformation (FRSST) is proposed. First, the theoretical derivation of the FRSST algorithm as a signal processing technique is given and the steps of the IF estimation are presented. Second, the main advantages of the proposed FRSST algorithm are proved. In the comparison of simulation signals, it is verified that the FRSST algorithm has a high time-frequency concentration, is non-fragile to the frequency modulation rate, has noise robustness and has nonsensitivity to the cross-frequency signal. Finally, the FRSST algorithm is applied to the IF estimation of a rolling bearing under rising speed and fluctuated speed, and is compared with the SST based on variational mode decomposition (VMD-SST), the generalized parametric SST (PSST) and polynomial chirplet transform (PCT). The test results show that the estimation error of IF based on the FRSST method is the least for a rolling bearing with the four fault types under rising speed. On average, the estimation error is 2.2180 Hz less than the corresponding error of the VMD-SST and 1.1862 Hz less than the corresponding error of the PSST method. INDEX TERMS Fractional synchrosqueezing transformation, instantaneous frequency, noise robustness, rolling bearing.