Defect Inspection of Textured Surfaces is a challenging problem which occurs during manufacturing in many processing phases. With arbitrary length, shape and orientation, these defects occur. Moreover, there are fewer and different photos of defective products. Deep Convolution Neural Network (CNN) has an impressive development in target detection, and better results have been obtained with the implementation of deep CNN design for texture detection. Nonetheless, with the growing detection accuracy of deep CNNs, there are the drawbacks of significantly increasing computational costs and processing resources, which seriously hinders CNN's use in resource-limited environments such as mobile or embedded phones. In this paper, a novel framework is proposed that uses raw image database patch statistics joint with two layers of neural network for surface defect detection. For defect detection, a convolution neural network (CNN) classifier is used. Imaging analysis of training samples using Deep Convolution Neural Network (CNN) is used to find the defect in an image's target area. In point of energy saving, the results of the experiment show that proposed method has numerous advantages in terms of reduction in time and cost. It also shows the high-performance contrast to conventional manual inspection process with less repetition and helps to build the object detection classifier with high generalization potential and high detection accuracy.