Recently, researches for new waveforms have been carried in order to satisfy KPI for 5G mobile communication system. UFMC and FBMC are well known as ways to reduce OOB power. In this paper, we analyze spectrum of OFDM, UFMC, and FBMC system under nonlinear HPA environment. And then we evaluate and analyze performance of these systems under the same environment. As simulation results, OOB power of OFDM, UFMC, and FBMC increases with HPA nonlinearity increasement. Especially, OOB power of FBMC is the largest increase due to HPA nonlinearity. Also, BER performance of these systems is degraded by increasement of HPA nonlinearity. Therefore, In this paper, we can confirm that a system with low-complexity and high-speed processing is advantageous, because BER performance and OOB power characteristic of these systems are degraded to similar level, when these systems are implemented in real world.

In this paper, we have proposed MIMO-IBFD system that can transmit and receive multiple data stream at the same time in the single band, and analyze the system. We have confirmed that MIMO-IBFD system requires multiple RF cancellation and multiple Digital cancellation techniques. In simulation, we have analyzed characteristic of received signal on MIMO-IBFD system, and evaluate system performance of conventional 2x2 MIMO system and MIMO-IBFD system. As simulation results, we have confirmed that MIMO-IBFD system can transmit and receive multiple data stream at the same time in the single band by using multiple RF and digital cancellation techniques. Additionally, we have confirmed that BER performance of 2x2 MIMO-IBFD system with low-level QAM modulation is similar to performance of 2x2 MIMO system, and BER performance of 2x2 MIMO-IBFD system with high-level QAM modulation is degraded in comparison with 2x2 MIMO system with high-level QAM modulation. We can confirm that MIMO-IBFD system requires high level self-interference cancellation technique in order to use high level modulation.