Mangifera Indica, commonly known as mangoes, is the most commercialized export fruit crop in India, accounting for about 40% of the total global production. Due to its widespread production, it is vulnerable to a variety of diseases that affect its yield and resulting in loss. These diseases like Anthracnose, Powdery Mildew, Leaf blights, etc., occur primarily on leaves. As a result, there is a great need for a system that helps in the detection of diseased mango leaves. In this paper, we propose a system that makes use of pre-trained Convolutional Neural Network architecture, the ResNet-50 for the detection of infected mango leaves. The dataset contains 435 images of mango leaves with binary classification as healthy and diseased. These images are pre-processed by resizing them and applying CLAHE. After applying in-place data augmentation on the dataset, the features are extracted using the ResNet-50 model. For the classification process, we make use of fine-tuned head and Machine Learning classifiers such as Support Vector Machine, Gradient Boosting, Logistic Regression, XGBoost, Decision Tree, and K Nearest Neighbour. Among them, the fine-tuned head classifier achieved an accuracy of 97.7%, and Machine Learning classifiers such as SVM, Logistic Regression obtained an accuracy of 100%. The experimental results obtained validate that the system is efficient in its performance of detecting the two classes of mango leaves.