This paper presents passive vibration control methodology in internal turning process with the use of hybrid nanocomposite coatings (nano-SiC/GFRE) on the surface of boring bar. Natural frequencies and damping ratio of different composition tool holders are obtained experimentally using impact hammer test. Three different configuration considered are: conventional (tool holder 1); nano-SiC/GFRE with 1% SiC (tool holder 2); and nano-SiC/GFRE with 2% SiC (tool holder 3). A better damping ability is noticed in third configuration of tool holder compared to others. Furthermore, using single mode data, analytical stability lobe diagrams are constructed for all three tool holders. Moreover, Box-Behnken design (BBD) is adopted and a set of fifteen experiments are performed with each tool holder. For third configuration of tool holder, effect of input variables on the surface roughness and tool vibration amplitudes is studied using neural network model. Finally, the neural network regression model is employed as a function estimation tool in simulated annealing for obtaining optimal cutting conditions.