Quality control procedures are fundamental to any manufacturing process that intend to ensure that the product adheres to a defined set of requirements. However, manual quality control procedures tend to be visually laborious, tedious, and vulnerable to human mistakes. To meet the ever-growing demand for high-quality products, the use of intelligent visual inspection systems is gaining importance for deployment in production lines. Many works imbibing image processing techniques, machine learning, and neural network models have been proposed to perform defect detection and segmentation focused on specific domains of defects. Defects in manufacturing processes manifest in varied forms and attributes which add to the woes of developing a one-shot methodology for defect detection, while it is also very expensive to generate a dataset of images capturing the variety to train a one-shot machine learning model. This paper presents a framework that captures the essence of defect detection by proposing a mind-map to classify various defects based on visual attributes, and another to classify the various processing methodologies presented in the literature. In addition, the paper proposes a mapping between the class of defects and methodologies to act as a basis for anyone to come up with a solution for defect detection in their use cases. It also proposes an empirical recommendation formula, based on three image metrics to judge the performance of a method over a given class of images. This paper showcases the implementation of a Smart Defect Segmentation Toolbox assimilating methodologies like Wavelet Analysis, Morphological Component Analysis (MCA), Basic Line Detector (BLD), and presents case studies to support the working of the recommendation formula.