With the increase in academic pressure on college students, some students show signs of academic burnout, which may lead to mental health problems. This study aims to explore the mental health status of college students under the manifestation of academic burnout and to propose a mental health detection model based on electroencephalography (EEG) signals. In this study, college students with the manifestation of study burnout were selected as research subjects, and their EEG signals were collected to identify and analyze their psychological state using a mental health recognition model. In the selection of mental health recognition models, this study used a variety of classical models for comparative analysis, including support vector machine (SVM), k-nearest neighbor (KNN), logistic regression (LR), convolutional neural network (CNN), long and short-term memory network (LSTM), and convolutional neural network-long and short-term memory network (CNN-LSTM). Experimental results show that the CNN-LSTM model performs best in identifying students with mental health problems, and its accuracy and stability are better than other models. Through this study, we conclude that academic burnout is closely related to college students’ mental health problems, and EEG signals combined with the CNN-LSTM model can effectively identify the mental health status of these students. This finding provides a new method and tool for monitoring college students’ mental health, which helps detect and intervene in students’ mental health problems early and promote their healthy growth and development.