Brain-computer interface (BCI) research has gained attention in education contexts, offering the potential to monitor and enhance student’s cognitive states and this study focuses on developing an optimal deep learning model, ODL-BCI, for real-time classification of students’ concentration levels. The model incorporates hyper-parameter tuning techniques and utilizes the publicly available “Confused student EEG brainwave data” dataset. We propose a deep learning model with hyperparameters optimized through Bayesian optimization. The model’s architecture is constructed with an input layer, several hidden layers, and an output layer. The number of nodes and activation functions in the hidden layers is determined using Bayesian optimization. The learning rate is also optimized for each layer. The proposed model is evaluated and compared with several standard machine learning classifiers, including Decision Tree, AdaBoost, Bagging, MLP, Naïve Bayes, Random Forest, SVM, and XG Boost, on an EEG confusion dataset. Experimental results demonstrate that the optimized deep learning model outperforms all other classifiers, achieving an accuracy of 74 percent. The model’s effectiveness in accurately classifying students’ concentration levels highlights its potential as a valuable tool in educational settings. This research contributes to the advancement of BCI technology, providing insights into the optimization of deep learning models for EEG-based cognitive assessment. Future work involves exploring the model’s generalizability on larger datasets and extending its applicability to other BCI applications.