A novel nonlinear energy sink (NES) consisting of permanent magnetic springs and coil springs is proposed, and the vibration attenuation performance of the NES for unbalanced rotor system is investigated. Firstly, the nonlinearity of the magnet spring is analyzed and the structure of the NES is introduced. Then, the dynamic model of the rotor system with the NES is built, and the responses and stabilities of the system are studied by applying Complexification-averaging method. The strongly modulated responses (SMR) behavior, which is the most important performance characteristic of the NES, is analytically studied by combining Complexification-averaging method and multiscale method and numerically verified by Runge-Kutta method. The results show that the NES is effective in attenuating the vibration of unbalanced rotor, and the SMR occurrence range can be broadened by increasing the nonlinearity of the NES. And also, the NES has better performance over a wider frequency range than the linear absorber.