A high level of mathematical education is often associated with high effectiveness in solving cognitive problems and professional success. It is known that cognitive processes are accompanied by specific bioelectric activity in the brain and success in mathematical education as a behavioral phenotype is also reflected in EEG both during mental activity and at rest. This study tested the potential to distinguish volunteers with an advanced level of education in mathematics (AM) from individuals with a basic level of education in mathematics (BM) based on the frequency parameters of the resting-state electroencephalogram (EEG) recorded before the start of cognitive tasks. Further, the volunteers were divided into two groups, highly successful and moderately successful, according to their task-solving performance. The Light Gradient Boosting Machine learning algorithm was used for cross-subject classification based on the power spectral density of seven EEG frequency bands. It most accurately recognized and differentiated EEG of highly successful from highly successful BM subjects. The results indicate that success in solving tasks in combination with a high level of education in mathematics can be reflected in or predicted by the specific rhythmic activity of the brain at rest.