Cognitive radio networks face challenges from malicious users who aim to disrupt decision-making during spectrum sensing and decrease spectrum efficiency. The proposed model seeks to distinguish between the presence and absence of primary user signals using energy detection and also it introduces a quantum-inspired outlier detection mechanism for cognitive radio networks to enhance security and reliability during spectrum sensing by ignoring the identified outliers. It employs BB84 quantum key distribution (QKD) protocol and one time pad for secure transmission of secondary user (SU) local decisions to the fusion center (FC), ensuring data integrity and confidentiality. These decisions are transmitted securely establishing a shared secret key between SU and FC and arrives at a global decision. To make the model more robust, we proposed another novel model with a modified BB84 protocol and compared the results of both the models, offering a robust solution for enhancing the reliability and performance of cognitive radio networks with proven results from simulation.