Epilepsy is one of the most debilitating neurological diseases that abruptly alters a person's way of life. Manual diagnosis is a laborious and time-consuming task prone to human error; therefore, automating this task by developing an intelligent system is necessary. Existing deep learning (DL) models require high training time, large datasets, and machines with more memory and processing power. In addition, owing to the black-box nature of DL models, no one can determine the features that the network prefers for classification decisions. To overcome these challenges, this paper proposes an accurate, automatic, and fast intelligent system for epilepsy detection using a computer-aided diagnosis (CAD)-2D machine learning (ML) framework. Existing ML models struggle to produce reliable and acceptable diagnostic results owing to the low amplitude and nonstationary nature of electroencephalograms (EEG), particularly in clinical situations where environmental influences are almost impossible to eliminate. The proposed model was built on the CHB-MIT dataset, and it represents the first study that employs the speeded-up robust feature (SURF) bag of features (BOF) technique for this application, which generates local features from the spectrogram images of the respective 1D EEG signal inputs. In addition, DL features were extracted from the spectrogram images for model performance comparisons. Both features were used separately to train the ML classifiers. Implementing SURF offers fast computation and makes the model invariant to distortions, noise, scaling, and so on. Therefore, the proposed model is more suitable for real-time applications, proposed ML framework provides an enhanced accuracy of 99.78% from the SVM-RBF classifier, along with 99.56% sensitivity, 100% specificity, and an error rate of 0.22%. The higher detection accuracy demonstrated the effectiveness of the proposed framework for medical disease diagnosis applications.