In this paper, the effect of nonlinearity on vibration of a rotating shaft passing through critical speed excited by nonideal energy source is investigated. Here, the interaction between a nonlinear gyroscopic continuous system (i.e. rotating shaft) and the energy source is considered. In the shaft model, the rotary inertia and gyroscopic effects are included, but shear deformation is neglected. The nonlinearity is due to large deflection of the shaft. Firstly, nonlinear equations of motion governing the flexural–flexural–extensional vibrations of the rotating shaft with nonconstant spin are derived by the Hamilton principle. Then, the equations are simplified using stretching assumption. To analyze the nonstationary vibration of the nonideal system, multiple-scale method is directly applied to the equations expressed in complex coordinates. Three analytical expressions that describe variation of amplitude, phase, and angular acceleration during passage through critical speed are derived. It is shown that Sommerfeld effect in specific range of driving torque occurs. Finally, effect of damping and nonlinearity on occurrence of Sommerfeld effect is investigated. It is shown that the linear model predicts the range of Sommerfeld effect occurrence inaccurately and, therefore, nonlinear analysis is necessary in the present problem.