With the increase of the centrifugal compressor capability, such as large scale LNG and CO2reinjection, the stability margin evaluation is crucial to assure the compressor work in the designed operating conditions in field. Improving the precision of parameter identification of stability is essential and necessary as well. Based on the time-varying characteristics of response vibration during the sine-swept process, a short-time Fourier transform (STFT) filter was introduced to increase the signal-noise ratio and improve the accuracy of the estimated stability parameters. A finite element model was established to simulate the sine-swept process, and the simulated vibration signals were used to study the filtering effect and demonstrate the feasibility to identify the stability parameters by using Multiple-Input and Multiple-Output system identification method that combines the prediction error method and instrumental variable method. Simulation results show that the identification method with STFT filter improves the estimated accuracy much well and makes the curves of frequency response function clearer. Experiment was carried out on a test rig as well, which indicates the identification method is feasible in stability identification, and the results of experiment indicate that STFT filter works very well.