As a special micro-motion feature of rotor target, rotational angular velocity can provide a discriminant basis for target classification and recognition. In this paper, the authors focus on an efficient rotational angular velocity estimation method of the rotor target is based on the combination of the time–frequency analysis algorithm and Hough transform. In order to avoid the problems of low time–frequency resolution and cross-term interference in short-time Fourier transform and Wigner–Ville distribution algorithm, a modified short-time fractional Fourier transform (M-STFRFT) is proposed to obtain the time-FRFT domain (FRFD)-frequency spectrum with the highest time–FRFD–frequency resolution. In particular, an orthogonal matching pursuit (OMP)-based algorithm is proposed to reduce the computational complexity when estimating the matched transform order in the proposed M-STFRFT algorithm. Firstly, partial transform order candidates are selected randomly from the complete candidates. Then, a partial entropy vector corresponding to partial transform order candidates is calculated from the FRFT results and utilized to reconstruct the complete entropy vector via the OMP algorithm, and the matched transform order can be estimated by searching minimum entropy. Based on the estimated matched transform order, STFRFT is performed to obtain the time–FRFD–frequency spectrum. Moreover, Hough transform is employed to obtain the energy accumulation spectrum, and the micro-Doppler parameter of rotational angular velocity can be estimated by searching the peak value from the energy accumulation spectrum. Both simulated data and measured data collected by frequency modulated continuous wave radar validate the effectiveness of the proposed algorithm.