This paper addresses a novel sliding mode control based on state observer for active magnetic bearing rotor system. Firstly, the state-space model of a radial AMB rotor system is established with considering unbalance disturbance and gyro effect for a vertical flywheel energy storage system. Then a sliding mode function and switching surface are constructed based on an observer. Meanwhile, a separation and decoupling strategy based on Finsler’s lemma is proposed. Through this method, the constraint relationship between the controller gain, active magnetic bearing matrices and the Lyapunov variables is eliminated. After that a method for chattering reduction in the sliding-mode controller is raised. Relied on these techniques, new sufficient conditions for the stability of AMB rotor system are given in the framework of linear matrix inequalities. Finally, the effectiveness of the proposed sliding mode controller is validated on the experimental platform of the flywheel energy storage system.