Manual inspection of fruit diseases is a time-consuming and costly because it is based on naked-eye observation. The authors present computer vision techniques for detecting and classifying fruit leaf diseases. Examples of computer vision techniques are preprocessing original images for visualization of infected regions, feature extraction from raw or segmented images, feature fusion, feature selection, and classification. The following are the major challenges identified by researchers in the literature: (i) lowcontrast infected regions extract irrelevant and redundant information, which misleads classification accuracy; (ii) irrelevant and redundant information may increase computational time and reduce the designed model's accuracy. This paper proposed a framework for fruit leaf disease classification based on deep hierarchical learning and best feature selection. In the proposed framework, contrast is first improved using a hybrid approach, and then data augmentation is used to solve the problem of an imbalanced dataset. The next step is to use a pre-trained deep model named Darknet53 and fine-tune it. Next, deep transfer learning-based training is carried out, and features are extracted using an activation function on the average pooling layer. Finally, an improved butterfly optimization algorithm is proposed, which selects the best features for classification using machine learning classifiers. The experiment was carried out on augmented and original fruit datasets, yielding a maximum accuracy of 99.6% for apple diseases, 99.6% for grapes, 99.9% for peach diseases, and 100% for cherry diseases. The overall average achieved accuracy is 99.7%, higher than previous techniques.