Cognitive radio (CR) technology has emerged as a viable solution to solve the dilemma of spectrum scarcity and underutilization. It allows secondary users (SUs) to opportunistically access the primary user (PU) channel. Cooperative spectrum sensing (CSS) further improves the accuracy of detecting spectrum availability by exploiting spatial diversity gain. However, the open nature of CR networks (CRNs) makes CSS process be prone to Byzantine attack, resulting in the excessive interference to the PU and the waste of spectrum resources. To this end, we propose a joint spectrum sensing and resource allocation scheme to defend against Byzantine attack. Specifically, we develop a probabilistic Byzantine attack model to characterize attack behaviors from malicious users (MUs) and evaluate its negative impact on the CSS performance. Then, the delivery evaluation mechanism is designed to evaluate the sensing performance of SUs and distinguish between normal users and MUs. Based on this, we propose a selection-majority algorithm to improve CSS performance by exploiting sensing reports from MUs. To further solve the Byzantine attack, we propose a spectrum resource allocation algorithm, which allocates corresponding spectrum resources according to the historical behavior of SUs. By reducing the resources allocated to poorly performing SUs, this scheme incentivizes them to report correct sensing results and drive MUs to stop attacking or leave CRN. Extensive simulation results corroborate the correctness and effectiveness of our theoretical analysis and show that our proposed scheme can effectively resist Byzantine attack.