The output and quality of apples were greatly threatened by plant diseases. Identifying the types and grades of diseases in time was helpful to the management of diseases. When the disease occurs on a leaf, there was little change in the leaf except for the affected area. The traditional attention mechanism changed the weight of the network to all pixels in the image, which affected the ability of the network to extract the features of the lesion area. And the traditional method of data enhancement was easy to cause local similarity and local discontinuity of all sample features in the same disease grade. In this paper, the improved metric matrix for kernel regression (IMMKR) was used to reduce the influence of local similarity and local discontinuity of all sample features in the same class. Then, a new attention mechanism by fusing the lesion location based on visual features was proposed, and the attention of the model to the lesion area was strengthened. The experiments were carried out on three different diseases of apple named black rot, scab, and rust. The accuracy rate and recall rate of the new method on the combined dataset of PlantVillage and PlantDoc were 91.55% and 92.06%, respectively, which was superior to existing methods. This algorithm has important reference significance for the identification and promotion of crop diseases.