The tooth-slot transition creates an axial excitation on traveling wave ultrasonic motors. It induces a rotor’s axial rigid vibration, which in turn affects the contact state and arouses speed fluctuation. To gain an insight into this problem, this work examines the relationships between the tooth-slot transition, axial vibration, nonlinear contact, and speed fluctuation. An analytical model governing rotor’s vibration is developed, where the transition force, nonlinear contact stiffness, and pre-pressure are included. The contact stiffness is demonstrated to decrease with an increase in stator’s vibration amplitude and it is approximated by polynomial fitting such that the nonlinear problem can be analytically solved. The primary, 1/2 subharmonic, and 2/1 superharmonic resonances are analyzed to determine the amplitude–frequency response and steady-state response. Nonlinear phenomena regarding the three types of resonances are identified. The interaction between the transition force, axial vibration, contact stiffness and speed fluctuation is investigated through a new contact model. The results imply that the rotor vibration induces rotating acceleration and leads to speed fluctuation. In addition, the rotor speed fluctuates in a similar fashion with the axial vibration.