Due to its ability to boost the spectral efficiency of wireless communications systems, non-orthogonal multiple access (NOMA) has been deemed promising. NOMA retains the necessary effectiveness to enable 5G communication. The wireless network’s spectral efficiency and energy are reduced due to the limited spectrum and rising demands of users. Because of the mutual cross-tier interference that occurs in heterogeneous networks, NOMA presents brand-new technical difficulties in resource allocation. The use of non-orthogonal resources and spectrum sharing can cause interference that lowers the performance. Therefore, incorporating quality-of-service (QoS) into the design of a new NOMA model with improved bandwidth efficiency and energy efficiency (EE) is absolutely necessary. A deep learning strategy for maximizing the efficiency of spectrum and energy with QoS in NOMA is presented in this paper. In order to increase the efficiency of spectrum and energy with QoS in the NOMA system, an adaptive artificial rabbits Harris Hawks optimization (AARHHO) algorithm is developed to optimize parameters such as the time allocation ratio and beam forming vectors presented in the full-duplex (FD) relay and base station (BS). As a result, the NOMA network efficiency of bandwidth and energy is effectively maximized with QoS using the newly developed AARHHO approach.