Electroencephalogram (EEG) plays a significant role in emotion recognition because it contains abundant information. However, due to the highly correlated EEG channels, a lot of redundant EEG features exist, which not only potentially degrade the emotion recognition accuracy, but also bring high computational cost. To address this challenge, this paper proposes an adaptive matrix-based evolutionary computation (MEC) framework to select as a small number of informative EEG features as possible for effective emotion recognition. Unlike most existing EC algorithms that utilize vector-based operations, this framework leverages matrix-based operations to reduce feature redundancy and improve classification accuracy by dynamically adjusting the feature subset size according to the characteristics of the dataset. In such a way, the selection efficiency is largely improved. To verify the effectiveness and efficiency of this framework, the classical Genetic Algorithm (GA), the typical Particle Swarm Optimization (PSO) algorithm, and the classical Differential Evolution (DE) algorithm, are respectively embedded into this framework for EEG feature selection, and then evaluated on three widely used public EEG datasets for emotion recognition. Compared with several state-of-the-art EEG feature selection algorithms, the devised framework is much more effective in terms of the classification accuracy, the feature subset optimization, and the computational efficiency. In addition, the experimental results further reveal that the selected feature subsets are very different for different genders. This indicates the demand of gender-sensitive EEG feature selection for emotion recognition.