Cognitive radio is a promising technology for the future wireless spectrum allocation to improve the utilization rate of the licensed bands. However, the cognitive radio network is susceptible to various attacks. Hence, there arises a need to develop a highly efficient security measure against the attacks. This paper presents a beamforming-based feature extraction and relevance vector machine (RVM)-based method for the classification of the attacker nodes in the cognitive radio network. Initially, the allocation of the Rayleigh channel is performed for the communication. The quaternary phase shift keying method is used for modulating the signals. After obtaining the modulated signal, the extraction of the beamforming-based features is performed. The RVM classifier is used for predicting the normal nodes and attacker nodes. If the node is detected as an attacker node, then communication with that node is neglected. Particle swarm optimization is applied for predicting the optimal channel, based on the beamforming feature values. Then, signal communication with the normal nodes is started. Finally, the signal is demodulated. The signal-to-noise ratio and bit-error rate values are computed to evaluate the performance of the proposed approach. The accuracy, sensitivity, and specificity of the RVM classifier method are higher than the support vector machine classifier. The proposed method achieves better performance in terms of throughput, channel sensing/probing rate, and channel access delay.KEY WORDS: beamforming-based feature extraction; bit-error rate (BER); quaternary phase shift keying (QPSK); cognitive radio network (CRN); particle swarm optimization (PSO); relevance vector machine (RVM); signal-to-noise ratio (SNR)