Numerical simulation is an effective approach in studying cutting mechanism. The widely used methods for cutting simulation include finite element analysis and molecular dynamics. However, there exist some intrinsic shortcomings when using a mesh-based formulation, and the capable scale of molecular dynamics is extremely small. In contrast, smoothed particle hydrodynamics (SPH) is a candidate to combine the advantages of them. It is a particle method which is suitable for simulating the large deformation process, and is formulated based on continuum mechanics so that large scale problems can be handled in principle. As a result, SPH has also become a main way for the cutting simulation. Since some issues arise while using conventional SPH to handle solid materials, the total Lagrangian smoothed particle hydrodynamics (TLSPH) is developed. But instabilities would still occur during the cutting, which is a critical issue to resolve. This paper studies the effects of TLSPH settings and cutting model parameters on the numerical instability, as well as the chip formation process. Plastic deformation, stress field and cutting forces are analyzed as well. It shows that the hourglass coefficient, critical pairwise deformation and time step are three important parameters to control the stability of the simulation, and a strategy on how to adjust them is provided.