The drive of this study is to develop a robust system. A method to classify brain magnetic resonance imaging (MRI) image into brain‐related disease groups and tumor types has been proposed. The proposed method employed Gabor texture, statistical features, and support vector machine. Brain MRI images have been classified into normal, cerebrovascular, degenerative, inflammatory, and neoplastic. The proposed system has been trained on a complete dataset of Brain Atlas‐Harvard Medical School. Further, to achieve robustness, a dataset developed locally has been used. Extraordinary results on different orientations, sequences of both of these datasets as per accuracy (up to 99.6%), sensitivity (up to 100%), specificity (up to 100%), precision (up to 100%), and AUC value (up to 1.0) have been achieved. The tumorous slices are further classified into primary or secondary tumor as well as their further types as glioma, sarcoma, meningioma, bronchogenic carcinoma, and adenocarcinoma, which could not be possible to determine without biopsy, otherwise.