In this manuscript, a deep learning approach is used to carry out research on wheat seed variety identification, and a fast and efficient wheat seed variety identification method (IRB-5-CA Net) is proposed based on the characteristics of wheat seeds and a self-constructed dataset, which provides ideas for wheat seed variety identification. Twenty-nine wheat varieties grown under natural light conditions were selected as the research objects, and a wheat seed dataset with the number of 4,385 sheets was constructed by integrating sunny, cloudy, and rainy conditions with a blue hard paper as the background plate, and using a Nikon COOLPIX B700 digital camera for dataset collection. Based on the above self-constructed dataset, by improving the MobileNetV2 model, a new lightweight method (IRB-5-CA Net) for wheat seed recognition was proposed. IRB-5-CA Net specific improvements are listed below: adding 5×5 convolution to the bottleneck without using the shortcut structure and adding the Coord Attention to the bottleneck with using the shortcut structure. After training IRB-5-CA Net on the self-built dataset, the average accuracy, average recall, and F1 values are 99.5%, 99.6%, and 99.6%. The model improves the average accuracy by 6.8%, 5.6%, 5.8%, and 8.3% compared to MobileNetV2, ResNet34, Efficientnetv2_s, and GoogLeNet. The IRB-5-CA Net was visualised using the pytorch_grad_cam method, in the output heat map, it can be seen that the model focuses more attention on wheat seeds, resulting in higher accuracy. Applying IRB-5-CA Net to other public datasets such as wheat seed disease, apple leaf disease, and AI Challenger 2018 crop disease detection, the average accuracy was 98.06%, 96.15%, and 94.02%. This study provides a theoretical basis for seed variety identification, disease identification, and other crop disease identification in wheat.