A brain tumor is essentially a collection of aberrant tissues, so it is crucial to classify tumors of the brain using MRI before beginning therapy. Tumor segmentation and classification from brain MRI scans using machine learning techniques are widely recognized as challenging and important tasks. The potential applications of machine learning in diagnostics, preoperative planning, and postoperative evaluations are substantial. Accurate determination of the tumor’s location on a brain MRI is of paramount importance. The advancement of precise machine learning classifiers and other technologies will enable doctors to detect malignancies without requiring invasive procedures on patients. Pre-processing, skull stripping, and tumor segmentation are the steps involved in detecting a brain tumor and measurement (size and form). After a certain period, CNN models get overfitted because of the large number of training images used to train them. That is why this study uses deep CNN to transfer learning. CNN-based Relu architecture and SVM with fused retrieved features via HOG and LPB are used to classify brain MRI tumors (glioma or meningioma). The method’s efficacy is measured in terms of precision, recall, F-measure, and accuracy. This study showed that the accuracy of the SVM with combined LBP with HOG is 97%, and the deep CNN is 98%.