Currently, mental stress is a common social problem affecting people. Stress reduces human functionality during routine work and may lead to severe health defects. Detecting stress is important in education and industry to determine the efficiency of teaching, to improve education, and to reduce risks from human errors that might occur due to workers’ stressful situations. Therefore, the early detection of mental stress using machine learning (ML) techniques is essential to prevent illness and health problems, improve quality of education, and improve industrial safety. The human brain is the main target of mental stress. For this reason, an ML system is proposed which investigates electroencephalogram (EEG) signal for thirty-six participants. Extracting useful features is essential for an efficient mental stress detection (MSD) system. Thus, this framework introduces a hybrid feature-set that feeds five ML classifiers to detect stress and non-stress states, and classify stress levels. To produce a reliable, practical, and efficient MSD system with a reduced number of electrodes, the proposed MSD scheme investigates the electrodes placements on different sites on the scalp and selects that site which has the higher impact on the accuracy of the system. Principal Component analysis is employed also, to reduce the features extracted from such electrodes to lower model complexity, where the optimal number of principal components is examined using sequential forward procedure. Furthermore, it examines the minimum number of electrodes placed on the site which has greater impact on stress detection and evaluation. To test the effectiveness of the proposed system, the results are compared with other feature extraction methods shown in literature. They are also compared with state-of-the-art techniques recorded for stress detection. The highest accuracies achieved in this study are 99.9%(sd = 0.015) and 99.26% (sd = 0.08) for identifying stress and non-stress states, and distinguishing between stress levels, respectively, using only two frontal brain electrodes for detecting stress and non-stress, and three frontal electrodes for evaluating stress levels respectively. The results show that the proposed system is reliable as the sensitivity is 99.9(0.064), 98.35(0.27), specificity is 99.94(0.02), 99.6(0.05), precision is 99.94(0.06), 98.9(0.23), and the diagnostics odd ratio (DOR) is ≥ 100 for detecting stress and non-stress, and evaluating stress levels respectively. This shows that the proposed framework has compelling performance and can be employed for stress detection and evaluation in medical, educational and industrial fields. Finally, the results verified the efficiency and reliability of the proposed system in predicting stress and non-stress on new patients, as the accuracy achieved 98.48% (sd = 1.12), sensitivity = 97.78% (sd = 1.84), specificity = 97.75% (sd = 2.05), precision = 99.26% (sd = 0.67), and DOR ≥ 100 using only two frontal electrodes.