Based on electroencephalography (EEG) and video data, we propose a multimodal affective analysis approach in this study to examine the affective states of university students. This method is based on the findings of this investigation. The EEG signals and video data were obtained from 50 college students experiencing various emotional states, and then they were processed in great detail. The EEG signals are pre-processed to extract their multi-view characteristics. Additionally, the video data were processed by frame extraction, face detection, and convolutional neural network (CNN) operations to extract features. We take a feature splicing strategy to merge EEG and video data to produce a time series input to realize the fusion of multimodal features. This allows us to realize the fusion of multimodal features. In addition, we developed and trained a model for the classification of emotional states based on a long short-term memory network (LSTM). With the help of cross-validation, the experiments were carried out by dividing the dataset into a training set and a test set. The model’s performance was evaluated with the help of four metrics: accuracy, precision, recall, and F1-score. When compared to the single-modal method of sentiment analysis, the results demonstrate that the multimodal approach, which combines EEG and video, demonstrates considerable advantages in terms of sentiment detection. Specifically, the accuracy obtained from the multimodal approach is significantly higher. As part of its investigation, the study also investigates the respective contributions of EEG and video aspects to emotion detection. It discovers that these features complement each other in a variety of emotional states and have the potential to improve the overall recognition results. The multimodal sentiment analysis method that is based on LSTM offers a high level of accuracy and robustness when it comes to recognizing the affective states of college students. This is especially essential for enhancing the quality of education and providing support for mental health.
