The increasing demand for seamless wireless services coupled with underutilization of limited spectrum due to fixed channel allocation policy have posed numerous challenges to wireless communications. There is need to design appropriate channel allocation strategies for continuous communication by mobile users for higher spectral efficiency. Cognitive radio network (CRN) offers a solution to the spectrum scarcity problem inherent in 4G and other networks through dynamic spectrum access, by allowing unlicensed secondary users (SUs) with cognitive devices to opportunistically access the spectrum holes when the licensed primary users (PUs) are not occupying them. Often times, measurements taken by SUs during the sensing process are uncertain due to multipath fading, shadowing and varying channel conditions. This results in imprecise spectrum detection and selection by SUs during switching decisions causing incessant spectrum handoff and undesirable ping-pong effect. In this paper, a support vector machine (SVM) classifier is used to categorize spectrum into two classes of busy and idle. Then, based on heterogeneous quality of service (QoS) requirements of the SUs, dynamic activities of PUs, and the fluctuating channel state information, a QoS-aware Adaptive Neuro-Fuzzy Inference System (ANFIS) framework is developed for spectrum switching decision using underlay spectrum access model. SVM predicted spectrum holes with 98.8% accuracy. ANFIS model yielded a 91.62% accuracy in the task of allocating spectrum holes to SUs for coexistent with PUs. Results further indicate that the intelligent framework can ensure fairness among SUs, reduce interference, improve throughput, and spectral efficiency. It can be deployed in disaster relief and emergency, public safety, and battlefield environments.