The fruit and vegetable classification problem is an inseparable branch in the field of image recognition. GoogLeNet provides a more ideal solution for the fruit and vegetable classification problem. We use the GoogLeNet network to classify apples, lemons, oranges, pomegranates, tomatoes, and colored peppers. Through experiments, we obtained the training accuracy of GoogLeNet as 96.88%, the testing accuracy as 96%, and the training speed as 11.38 sheets/second. The recognition accuracy of this model can meet the basic recognition requirements, but the training speed is low. Therefore, we decided to optimize GoogLeNet to significantly improve the training speed and further enhance the recognition accuracy of GoogLeNet. We reduced the number of convolutional kernels of GoogLeNet and adjusted the structure of Inception, which reduced the number of parameters of GoogLeNet by nearly 48% and increased the training speed of GoogLeNet from 11.38 to 33.68 sheets/second. To further improve its recognition accuracy, we tried two methods: 1) introducing a new activation function Swish; 2) between convolutional layers, we introduced DropBlock layer; these two methods improved the testing accuracy of GoogLeNet by 2%. Finally, we introduce AlexNet, VGGNet, ResNet18, DenseNet121, and Inception-ResNet to compare with our optimized GoogLeNet. By comparison, we found that our model has incredible performance in ACC, AUC, FPS, Recall, etc.