Ball-end milling cutters are one of the most widely used cutters in the automotive, aerospace, die and machine parts industries. In addition, milling chatter will reduce the surface quality and production efficiency, resulting in noise. It is particularly important to model the cutting force and analyze the flutter stability of ball-end milling cutters. In this paper, a simplified milling force model of ball-end milling cutter with three degrees of freedom was established based on Merchant bevel cutting theory. The model simplified the milling force coefficient. The expressions of instantaneous milling area considering the vibration displacements in X, Y and Z directions were derived. The nonlinear dynamic cutting force model of ball-end milling cutter with three degrees of freedom was established. The nonlinear chatter vibration mechanical model of ball-end milling cutter with three degrees of freedom was established by introducing the time delay term, the stability analysis is carried out by time domain simulation. The proposed models were experimentally verified.