Purpose In order to optimize BER and to substantiate performance measures, initially, the filter bank multicarrier (FBMC) quadrature amplitude modulation (QAM) performance metrics are evaluated with the cyclic prefix-orthogonal frequency division multiplexing (CP-OFDM) system. The efficiency of CP-OFDM, as well as FBMC/QAM that is transmitting over specific fading channels, is evaluated in terms of quality trade-off metrics over bit error rate (BER) as well as modulation order. When compared with the traditional FBMC systems, the proposed FBMC QAM system shows better performance. The performance metrics of FBMC/QAM with the inclusion of multiuser multiple-input-multiple-output (MUMIMO) is validated with worst case channel environment. The performance penalty gap that exists in CP- OFDM is compared with improved FBMC QAM in terms of both BER and OOB radiation measures. The BER trade off comparison between ML and MMSE optimally determine the prominent signal detection model for high performance FBMC QAM system. Design/methodology/approach The main objective of this research work is to provide perceptions about performance, co-channel interference avoidance as well as about the techniques that are used for minimizing the complexity of the system that is related to FBMC QAM structure for reducing intrinsic interference with higher spectral features as well as maximal likelihood (ML) detector systems. Findings This research work also looks at the efficiency of multiuser multiple-input-multiple-output (MU-MIMO) FBMC/QAM over nonlinear channels. Furthermore, when compared with OFDM, it also significantly reduces the penalty gap efficiency, thereby enabling the accessibility of the proposed FBMC QAM system from BER as well as implementation point of view. Finally, the signal detection is facilitated by the sub-detector and is achieved on the downlink side by making use of threshold-driven statistical measures that accurately minimize the complexity trade-off measures of the ML detector over modulation order. The computation of the proposed FBMC method’s BER performance measures was carried out through MATLAB simulation environments, as well as efficiency of the suggested work was demonstrated through detailed analyses. Originality/value This research work intend to combine the efficient MU-MIMO based transmission scheme with optimal FBMC/QAM for improved QoS over highly nonlinear channels which includes both delay spread and Doppler effects. And optimal signal detection model is facilitated at the downlink side by making use of threshold-driven statistical measures that accurately minimize the complexity trade-off measures of the ML detector over modulation order. The computation of the proposed FBMC method’s BER performance measures was carried out through MATLAB simulation environments, as well as efficiency of the suggested work was demonstrated through detailed analyses.