Summary Enabling a reliable and secure communication in cognitive radio networks (CRNs) is one of the interesting and demanding research areas and these days, it is used in wide range of applications. The traditional works mainly aimed to improve the spectrum efficiency and bandwidth utilization of the CRN. Moreover, it focused on attack detection in the network for improving the reliability. Thus, this work intended to secure the CRN by considering the objectives of both attack detection and quality of service (QoS) improvement. For this purpose, an integrated digital signature and mutual authentication (IDsMA) mechanism is proposed in this paper, which incorporates the benefits of both security and reliability. Also, the Rivest‐Sharmir‐Adlemann (RSA) algorithm is utilized to encrypt and decrypt the data at the transmitter and receiver side. Here, the unique user ID is generated by the service eNodeB based on the hash function, which is validated at the server side for validating the authenticity of the user. Furthermore, the particle swarm optimization (PSO) and relevant vector machine (RVM) techniques are utilized for optimal channel selection and attack classification. The simulation results evaluate the performance of the proposed technique by using various performance measures. Also, its superiority is proved by comparing the proposed technique with the existing techniques.