Tumor grown in the human brains is one of the significant reasons that lead to loss of lives globally. Tumor is malignant collection of cells that grow in the human body. If these tumors grow in the brain, then they are called as brain tumors. Every year large number of human lives are lost due to this disease. Early detection of the disease might save the lives but requires experienced clinicians and diagnostic procedure that requires time and is very expensive. Therefore, there is a requirement for a robust system that automates the process of tumor identification. The idea behind this paper is to diagnose brain tumors by identifying the affected regions from the brain MRI images using machine learning approaches. In the proposed approach, prominent features of the tumor images are collected by passing them through a pre-trained Convolutional Network, VGG16. We observe that SVM gives better accuracy than other models. Though we achieve 84% accuracy, we feel the performance is not satisfactory. To make the model more robust, we obtain the most discriminant features, by applying Linear Discriminant Analysis (LDA) on the features obtained from VGG16. We use different conventional models like logistic regression, K-Nearest neighbor classifier (KNN), Perceptron learning, Multi Layered Perceptron (MLP) and Support Vector Machine (SVM) for the comparison study of the tumor image classification task. The proposed model leads to an accuracy of 100% as deep features extract important characteristics of the data and further LDA projects the data onto the most discriminant directions.