Traditional plant disease diagnosis methods are mostly based on expert diagnosis, which easily leads to the backwardness of crop disease control and field management. In this paper, to improve the speed and accuracy of disease classification, a plant disease detection and classification method based on the optimized lightweight YOLOv5 model is proposed. We propose an IASM mechanism to improve the accuracy and efficiency of the model, to achieve model weight reduction through Ghostnet and WBF structure, and to combine BiFPN and fast normalization fusion for weighted feature fusion to speed up the learning efficiency of each feature layer. To verify the effect of the optimized model, we conducted a performance comparison test and ablation test between the optimized model and other mainstream models. The results show that the operation time and accuracy of the optimized model are 11.8% and 3.98% higher than the original model, respectively, while F1 score reaches 92.65%, which highlight statistical metrics better than the current mainstream models. Moreover, the classification accuracy rate on the self-made dataset reaches 92.57%, indicating the effectiveness of the plant disease classification model proposed in this paper, and the transfer learning ability of the model can be used to expand the application scope in the future.