Optimization of machining parameters like cutting speed, feed, and depth of cut is one of the extensively studied fields in the past two decades. While researchers agree optimization of these parameters is essential, there is no conscience as to what the objective of the optimization should be. The studies consider production cost, production time, surface finish, among others, as the objective of parameter optimization, but there are very few studies that consider the manufacturer prescribed tool life as the criteria for parament optimization. Among the methods that do consider tool life as an optimization objective, very few are closed-loop systems and these systems are facing challenges to generalizing when the application changes or the machining material changes or the tool geometry changes. Considering this, a novel image feedback using a convolution neural network-based method combined with principles of fuzzy logic is used to optimize machining parameters. Since the system is based on online feedback from the images of the inserts, it can be used for different materials, and the system is invariant to the different tool geometries and grades as the decisions are based on the wear mechanisms detected. The hybrid system is validated through experimentation for the turning application, but the methodology can be easily adapted for other machining applications.