Surface defect detection based on machine vision and convolutional neural networks (CNNs) is an important and necessary process that enables rubber ring manufacturers to improve production quality and efficiency. However, such automatic detection always consumes substantial computer resources to guarantee detection accuracy. To solve this problem, in this paper, a CNN optimization algorithm based on the Ghost module is presented. First, the convolutional layer is replaced with the Ghost module in CNNs so that feature maps can be generated using cheaper linear operations. Second, an optimization method is used to obtain the best replacement of the Ghost module to balance computer resource consumption and detection accuracy. Finally, an image preprocessing method that includes inverting colors is applied. This algorithm is integrated into YOLOv5, trained on a dataset of rubber ring surface defects. Compared to the original network, the network size decreases by 30.5% and the computational cost decreases by 23.1%, whereas the average precision only decreases by 1.8%. Additionally, the network’s training time decreases by 16.1% as a result of preprocessing. These results show that the proposed approach greatly helps practical rubber ring surface defect detection.