In this paper, the vibration characteristic of the casing under rub-impact condition is analyzed by using finite element method, envelope demodulation and empirical mode decomposition. Firstly, aiming at the structure characteristics of the casing of the rotor system, the finite element model is established by ANSYS software. Modal analysis and transient analysis are carried out in detail. The first six order natural frequencies and corresponding vibration modes are acquired in the mode analysis. The transient response history and time-amplitude curves are obtained in the harmonic response analysis. The vibration characteristics and rule of the casing under rub-impact condition are also presented in the transient dynamic analysis. Secondly, the vibration mechanism analysis of the casing is performed. Since the pulse force due to rubbing fault may lead to casing's resonance, the envelope demodulation method and empirical mode decomposition (EMD) are utilized to diagnose the rubbing fault. By means of the envelope demodulation and EMD, the stator's vibration signal is analyzed, and the rub-impact fault diagnosis is fulfilled. Finally, as for the aero-engine, the vibration signal obtained is just the acceleration on casing, and it is of great difficulty to obtain the vibration displacement. Hence, experiment of one-point rub-impact is conducted, and the validity and feasibility of the method adopted are verified by the testing response data. Besides, the casing vibration acceleration signals are analyzed in order to extract the rubbing fault's features. Results demonstrate that the casing resonance envelope signal is very suitable for detecting the rub-impact fault. It is worth relying that envelope demodulation method and EMD would have a promising application for the rubbing fault diagnosis between the rotor and the casing.