Surface defect detection is a crucial step in ensuring the quality of lenses. One method to check for surface defects is to use an optical system integrated with an industrial camera to magnify and highlight the position of a defect on the surface of a lens. Therefore, automatic optical inspection systems are applied to detect micro-defects. In this study, we propose an automatic inspection platform based on a deep neural network for automatically imaging and examining the surface of a lens. High-resolution images of 2448 × 2048 pixels are acquired using a hybrid lighting system. A convolutional neural network integrated with a trainable Gabor filter is used as a machine vision algorithm to perform image classification and defect segmentation tasks. The experimental results show that the proposed method effectively performed with noise in the background, achieving a segmentation accuracy of 98%.