The 5G system-based cognitive radio network is promised to meet the requirements of huge data applications with spectrum. However, the M-ary effect on the detection has not been thoroughly investigated. In this paper, an M-ary of quadrature amplitude modulation detection system is studied. Many rates are used in this study 4, 16, 64, and 256 constellation points. The detection system is applied to cooperative spectrum sensing to enhance the performance of detection for various rates of M-ary with low signal-to-noise ratio (SNR). Further, three kinds of signals based 5G system are sensed: filtered-orthogonal frequency division multiplexing (F-OFDM), filter bank multi-carrier (FBMC), and universal filtered multi-carrier (UFMC). The best detection performance is obtained when the M-ary=4 and number of SUs=50 user, whereas the worst detection performance is obtained when the M-ary=256 and number of SUs=10 user, as revealed in the simulation results. In addition, the detection performance for the F-OFDM signal is better than that of UFMC and FBMC signals for SNR <0 dB.