Considering the influence caused by joint angle, nonlinear damping, and nonlinear rigidity, the nonlinear torsional vibration dynamical modeling of the multi-DOF rolling mill’s main drive system is established. To analyze the coupled equations by analytic method, the equations are decoupled by transforming them into principal coordinates. The amplitude-frequency characteristic equations are obtained by multiscale method. Furthermore, numerical example based on the 1780 rolling mill’s main drive system of some Steel Co. is given to illustrate the effects of the resonance on the response of the system. The relationship between amplitude and frequency varies according to the parameters changes of nonlinear stiffness, nonlinear friction damping, torque disturbance, and joint angle. During the rolling process, the limited joint angles range is obtained and the variation rules of the joint angle caused by the nonlinear damping, nonlinear stiffness, and the disturbance torque are gained. The results present that the rolling mill can work more stably with the joint angle at a range from 2° to 8° by controlling the value of parameters. The research results provide a theoretical basis and reference for analyzing torsional vibration of rolling mill’s transmission system caused by joint angle.