Early diagnosis of brain tumors is essential for successful treatment and mortality prevention. The standard method for diagnosing brain tumors is magnetic resonance imaging. Due to the intricate structure of the brain's anatomy, it was difficult to segment and categorize the various types of brain tumors. Moreover, brain MRI contains a significant number of redundant or irrelevant features, classification algorithms struggle to effectively discover patterns in data not having the use of a feature selection algorithm. In this work, we propose a new wrapper feature selection technique for brain tumor classification. Before segmentation, pre-processing is first applied to the input images. The tumor is segmented using active contours, which are driven by laplacian of gaussian energy and optimal region scalable fitting energy. Hand-crafted features using the Histogram of oriented gradient, Gray level co-occurrence matrix, Gray level run length matrix, and deep features using the ResNet50 model are extracted from the segmented brain images. An improved ant lion optimization method (IALO) is proposed for the selection of the best features. Lastly, we classified brain tumors by using Support Vector Machines, Naive Bayes, and Recurrent Neural Networks using two publicly available datasets. The proposed method achieves an accuracy of 99.59% for BRATS and 99.18% for J.cheng brain images respectively. The proposed algorithm's performance was further compared with existing swarm intelligence-based metaheuristic optimization algorithms. From the experimental results, it is observed that the proposed IALO-based RNN method outperforms the existing methods and effectively removes the least significant features while improving classification accuracy.