A nonlinear dynamic model of spindle-cutter coupling system under cutting force, which takes into account the cutter stiffness and the nonlinearity of bearing clearance, is established. Analysis of the cross-section of a two-flute end mill is conducted to determine cutter stiffness. Then, the calculated cutter stiffness is introduced into the nonlinear dynamic model of coupling system. In the modeling of cutting force, the cutting width takes into account the cutter tip displacement. Moreover, nonlinear dynamic characteristics of spindle-cutter coupling system are studied and the effects of bearing clearance on the response of cutter tip are discussed as well, considering unbalanced force. The numerical results show that the bearing clearance strongly affects the equilibrium position. With different values of bearing clearance and rotation speed, the responses of cutter tip exhibit periodic, quasi-periodic and chaotic characteristics. Dynamic characteristics of spindle-cutter system depends on the bearing clearance and rotation speed. The proper bearing clearance and rotation speed should be chosen to ensure a stable cutting and high cutting rate according to the bifurcation diagram. The response of the cutter tip is a quasi-periodic motion when the cutting force is considered. The time-domain response of cutter tip predicted by nonlinear dynamic analysis can provide the basis for machining error prediction.