The green plum is rich in amino acids, lipids, inorganic salts, vitamins, and trace elements. It has high nutritional value and medicinal value and is very popular among Chinese people. However, green plums are susceptible to collisions and pests during growth, picking, storage, and transportation, causing surface defects, affecting the quality of green plums and their products and reducing their economic value. In China, defect detection and grading of green plum products are still performed manually. Traditional manual classification has low accuracy and high cost, which is far from meeting the production needs of green plum products. In order to improve the economic value of green plums and their products and improve the automation and intelligence level of the product production process, this study adopted deep learning methods based on a convolutional neural network and cost-effective computer vision technology to achieve efficient classification of green plum defects. First, a camera and LEDs were used to collect 1240 green plum images of RGB, and the green plum experimental classification standard was formulated and divided into five categories, namely, rot, spot, scar, crack, and normal. Images were randomly divided into a training set and test set, and the number of images of the training set was expanded. Then, the stochastic weight averaging (SWA) optimizer and w-softmax loss function were used to improve the VGG network, which was trained and tested to generate a green plum defect detection network model. The average recognition accuracy of green plum defects was 93.8%, the test time for each picture was 84.69 ms, the recognition rate of decay defect was 99.25%, and the recognition rate of normal green plum was 95.65%. The results were compared with the source VGG network, resnet18 network, and green lemon network. The results show that for the classification of green plum defects, the recognition accuracy of the green plum defect detection network increased by 9.8% and 16.6%, and the test speed is increased by 1.87 and 6.21 ms, respectively, which has certain advantages.