Breast cancer is the most common cancer diagnosed in women and the leading cause of cancer-related deaths among women worldwide. The death rate is high because of the lack of early signs. Due to the absence of a cure, immediate treatment is necessary to remove the cancerous cells and prolong life. For early breast cancer detection, it is crucial to propose a robust intelligent classifier with statistical feature analysis that considers parameter existence, size, and location. This paper proposes a novel Multi-Stage Feature Selection with Binary Particle Swarm Optimization (MSFS–BPSO) using Ultra-Wideband (UWB). A collection of 39,000 data samples from non-tumor and with tumor sizes ranging from 2 to 7 mm was created using realistic tissue-like dielectric materials. Subsequently, the tumor models were inserted into the heterogeneous breast phantom. The breast phantom with tumors was imaged and represented in both time and frequency domains using the UWB signal. Consequently, the dataset was fed into the MSFS–BPSO framework and started with feature normalization before it was reduced using feature dimension reduction. Then, the feature selection (based on time/frequency domain) using seven different classifiers selected the frequency domain compared to the time domain and continued to perform feature extraction. Feature selection using Analysis of Variance (ANOVA) is able to distinguish between class-correlated data. Finally, the optimum feature subset was selected using a Probabilistic Neural Network (PNN) classifier with the Binary Particle Swarm Optimization (BPSO) method. The research findings found that the MSFS–BPSO method has increased classification accuracy up to 96.3% and given good dependability even when employing an enormous data sample.