The current sorting process for winter jujubes relies heavily on manual labor, lacks uniform sorting standards, and is inefficient. Furthermore, existing devices have simple structures and can only be sorted based on size. This paper introduces a method for detecting surface defects on winter jujubes using convolutional neural networks (CNNs). According to the current situation in the winter jujube industry in Zhanhua District, Binzhou City, Shandong Province, China, we collected winter jujubes with different surface qualities in Zhanhua District; produced a winter jujube dataset containing 2000 winter jujube images; improved it based on the traditional AlexNet model; selected a total of four classical convolutional neural networks, AlexNet, VGG-16, Inception-V3, and ResNet-34, to conduct different learning rate comparison training experiments; and then took the accuracy rate, loss value, and F1-score of the validation set as evaluation indexes while analyzing and discussing the training results of each model. The experimental results show that the improved AlexNet model had the highest accuracy in the binary classification case, with an accuracy of 98% on the validation set; the accuracy of the Inception V3 model reached 97%. In the detailed classification case, the accuracy of the Inception V3 model was 95%. Different models have different performances and different hardware requirements, and different models can be used to build the system according to different needs. This study can provide a theoretical basis and technical reference for researching and developing winter jujube detection devices.