This paper presents a method to enhance computational efficiency of the nonlinear dynamic analysis of the large-scale deep-hole drilling machine. Based on finite element model, the drilling shaft system is constructed into Timoshenko beam element on the basis of flexible rotary shaft so as to increase the accuracy of numerical calculation. In order to save the calculation time and resources, modal synthesis technique is adopted to reduce the feature modal of linear freedom degrees of drilling shaft system. As a result, the accuracy required by the non-linear analysis will not be loss. On the basis of these, the whirling characteristics of drilling shaft system are studied under the conditions of different shaft lengths, and simultaneously, the stability patterns of drilling shaft motion and its stability region are obtained in the selected drilling depth and cutting speed parameters while drilling intersection holes.