A Brain-computer interface (BCI) using an electroencephalogram (EEG) signal has a great attraction in emotion recognition studies due to its resistance to humans’ deceptive actions. This is the most significant advantage of brain signals over speech or visual signals in the emotion recognition context. A major challenge in EEG-based emotion recognition is that a lot of effort is required for manually feature extractor, EEG recordings show varying distributions for different people and the same person at different time instances. The Poor generalization ability of the network model as well as low robustness of the recognition system. Improving algorithms and machine learning technology helps researchers to recognize emotion easily. In recent years, deep learning (DL) techniques, specifically convolutional neural networks (CNNs) have made excellent progress in many applications. This study aims to reduce the manual effort on features extraction and improve the EEG signal single model’s emotion recognition using convolutional neural network (CNN) architecture with residue block. The dataset is shuffle, divided into training and testing, and then fed to the model. DEAP dataset has class 1, class 2, class 3, and class 4 for both valence and arousal with an accuracy of 90.69%, 91.21%, 89.66%, 93.64% respectively, with a mean accuracy of 91.3%. The negative emotion has the highest accuracy of 94.86% fellow by neutral emotion with 94.29% and positive emotion with 93.25% respectively, with a mean accuracy of 94.13% on the SEED dataset. The experimental results indicated that CNN Based on residual networks can achieve an excellent result with high recognition accuracy, which is superior to most recent approaches.