In this paper, primary resonances of a flexible rotor suspended by active magnetic bearings are investigated. The rotor has large amplitude vibrations, which leads to nonlinearities in curvature and inertia. In addition, nonlinearity characteristics of active magnetic bearings are considered. In modeling of the shaft, rotary inertia and shear deformation are neglected but gyroscopic effect is included. Here, the equations of motion and related boundary conditions are both nonlinear. To analyze the dynamical behavior of the system, the renormalization group method is directly applied to the partial differential equations of motion and boundary conditions. The effects of different parameters of the shaft and active magnetic bearings on the frequency response of the system in the neighborhood of critical speeds are investigated, and it is shown that both hardening and softening nonlinearity behaviors are possible.