This present work examines the stability and nonlinear responses of a spindle milling system supported by ball bearings. A shaft finite element based on Timoshenko beam theory is employed to model the spindle, and modal reduction method is therefore adopted for saving the numerical calculating time. The issues of evaluating the effects of the ball bearing Hertz contact stiffness are consequently addressed. It is found that suitable constant bearing stiffness can be adopted to replace the nonlinear nonsmooth Hertz stiffness in prediction of the critical cutting depth of the milling system in certain bearing configuration conditions. For the constant bearing stiffness can be obtained by experiment, this replacement will undoubtedly simplify the spindle-bearing milling system. But with the increase in the bearing clearance, the spindle milling system will present obvious nonlinear behaviors, and the nonlinear Hertz contact bearing stiffness will take over. Isolated islands of chatter vibration, which are induced by the nonlinear nonsmooth bearing Hertz stiffness, can be found exist in milling processes in large bearing clearance conditions.