In order to investigate the mechanism of a rotor system with unbalance and blade-casing rubbing coupling faults, the vibration and rub force in a rotor system resulting from unbalance and blade-casing rubbing coupling faults are simulated. At first, a dynamic model of 0-2-1 form rotor is established, and a blade-casing model, which considered blade number, clearance between the blade tip and stator, stiffness and size of the blade, is introduced to the rotor model. The characteristics of unbalance and blade-casing rubbing coupling faults are analyzed by waterfall plot, spectrum, time domain chart of vibration and rub force, and the effects of speed and clearance between the blade tip and stator. Secondly, a rotor system tester is established according to the model structure. The experimental test of rubbing is simulated by adjusting the Feeding Device to control the clearance between the blade and stator. After simulation and experimental results were compared, it was found that fraction frequencies and high integer frequencies were affected by a blade-casing rubbing fault, the clearance and speed had different influence on the dynamic characteristic of rotor system with blade-casing rubbing fault.