Brain cancer deaths are significantly increased in all categories of aged persons due to the abnormal growth of brain tumor tissues in the brain. The death rate can be controlled by accurate early stage brain tumor diagnosis. The detection and classification of brain tumors play a crucial role in early diagnosis and treatment planning. Brain tumor detection and classification have become challenging and time-consuming for domain-specific radiologists and pathologists in medical image analysis. So, automatic detection and classification are essential to reduce the time of diagnosis. In recent years, machine learning classifiers have played an essential role in automatically classifying brain tumors. In this research, an approach based on an improved fuzzy factor fuzzy local information C means (IFF-FLICM) segmentation and hybrid modified harmony search and sine cosine algorithm (MHS-SCA) optimized extreme learning machine (ELM) is proposed for brain tumor detection and classification. The IFF-FLICM algorithm is utilized to accurately segment the brain’s magnetic resonance (MR) images to identify the tumor regions. The Mexican hat wavelet transform is employed for feature extraction from the segmented images. The extracted features from the segmented regions are fed into the MHS-SCA-ELM classifier for classification. The MHS-SCA is proposed to optimize the weights of the ELM model to improve the classification performance. Five distinct multimodal and unimodal benchmark functions are considered for optimization to demonstrate the robustness of the proposed MHS-SCA optimization technique. The image Dataset-255 is considered for this study. The quality measures such as SSIM and PSNR are considered for segmentation. The proposed IFF-FLICM segmentation achieved a peak signal-to-noise ratio (PSNR) of 37.24 dB and a structural similarity index (SSIM) of 0.9823. The proposed MHS-SCA-based ELM model achieved a sensitivity, specificity, and accuracy of 98.78%, 99.23%, and 99.12%. The classification performance results of the proposed MHS-SCA-ELM model are compared with MHS-ELM, SCA-ELM, and PSO-ELM models, and the comparison results are presented.