Automated brain tumor segmentation from reconstructed microwave (RMW) brain images and image classification is essential for the investigation and monitoring of the progression of brain disease. The manual detection, classification, and segmentation of tumors are extremely time-consuming but crucial tasks due to the tumor’s pattern. In this paper, we propose a new lightweight segmentation model called MicrowaveSegNet (MSegNet), which segments the brain tumor, and a new classifier called the BrainImageNet (BINet) model to classify the RMW images. Initially, three hundred (300) RMW brain image samples were obtained from our sensors-based microwave brain imaging (SMBI) system to create an original dataset. Then, image preprocessing and augmentation techniques were applied to make 6000 training images per fold for a 5-fold cross-validation. Later, the MSegNet and BINet were compared to state-of-the-art segmentation and classification models to verify their performance. The MSegNet has achieved an Intersection-over-Union (IoU) and Dice score of 86.92% and 93.10%, respectively, for tumor segmentation. The BINet has achieved an accuracy, precision, recall, F1-score, and specificity of 89.33%, 88.74%, 88.67%, 88.61%, and 94.33%, respectively, for three-class classification using raw RMW images, whereas it achieved 98.33%, 98.35%, 98.33%, 98.33%, and 99.17%, respectively, for segmented RMW images. Therefore, the proposed cascaded model can be used in the SMBI system.