The modeling of instantaneous rigid cutting force is the basis of physical simulation of milling. However, it is difficult to model due to the complex shape of the cutter teeth of ball-end milling cutter and the changeable posture during machining. Under the condition of considering the cutter posture adjustment, trajectory of cutter teeth is established based on the homogeneous coordinate transformation, an improved Z-MAP algorithm of recognition of instantaneous cutter and workpiece engagement area is proposed. By calculating the intersection point between the reference line of the cutter tooth and the boundary of the workpiece surface, or the one between the reference line and cutter teeth's sweeping surface, the instantaneous undeformed chip thickness is derived. And the cutting force coefficients are identified by non-linear regression method. On these bases, the calculation of the instantaneous cutting force model is established by using the infinitesimal integral method. In order to validate the reliability of the simulation model, vertical machining and inclined machining tests are carried out respectively. The results of experimental and simulated results are highly consistent, it is therefore indicated that the modeling simulation method is effective and could provide theoretical basis for the selection and optimization of parameters in actual machining.