The nonlinear dynamic analysis of rotating composite boring round bar containing carbon nanotubes (CNTs) in cutting system was investigated. Firstly, simplify the boring bar to the model of non-extendable rotary cantilever structure. Both the Halpin-Tsai model and micro-mechanical layering theory were used to predict the material properties of the boring round bar. Then the equations for the composite boring bar based on Euler Bernoulli shaft theory, including von Karman geometric nonlinearity are derived. The nonlinear dynamic model of cutting system including periodic regenerative chatter cutting force, periodic control force, viscoelastic and process damping is established by using Hamilton principle. The analytical solution of the steady-state response of the cutting system was subsequently obtained by the Galerkin approximation and the perturbation method of multiple time scales. Finally, the influences of carbon nanotube-related parameters, fiber volume fraction, fiber orientations, stacking sequences, damping coefficient and geometry properties of the cutter edge on the stability of the cutting system are evaluated. The obtained results show that the incorporation of CNTs has significant effect on the dynamic behavior of the cutting process. Increasing the process damping and changing the cutter edge and bar's cross section can improve stability of the cutting process. Furthermore, the unstable cutting region is sensitive to multi-valued properties generated by jumping.