Fifth-generation (5G) mobile systems are a necessary step toward successfully achieving further increases in data rates. As the use of higher-order quadrature amplitude modulation (QAM) is expected to increase data rates within a limited bandwidth, we propose a method for orthogonal frequency division multiplexing (OFDM)-based 1024-and 4096-QAM transmission with soft-decision Viterbi decoding for use in 5G mobile systems. Through evaluation of the transmission performance of the proposed method over multipath fading channels using link-level simulations, we determine the bit error rate (BER) performance of OFDM-based 1024-and 4096-QAM as a function of coding rate under two multipath fading channel models: extended pedestrian A (EPA) and extended vehicular A (EVA). We also demonstrate the influence of phase error on OFDM-based 1024-and 4096-QAM and clarify the relationship between phase error and the signal-to-noise ratio (SNR) penalty required to achieve a BER of 1 × 10 −2 . This work provides an effective solution for introducing higher-order modulation schemes in 5G and beyond.small-cells [8,9], which can improve both system capacity and link budget because each eNB in a small-cell is directly connected to a core network. As another small-cell strategy, the concept of a phantom cell, has been proposed to provide efficient management of radio resources and mobility operation, where small-cells are overlaid on a macro-cell [10,11]. This approach splits the control and user-data planes; that is, the macro-eNB manages all UE within the overlaid cells, although UE can connect to a neighboring low-power eNB in the small-cell.Heterogeneous networks (HetNets) are another solution of denser infrastructures in which smallor pico-cells with low-power eNBs are installed within the coverage area of a macro-cell [12][13][14][15][16]. The objective of HetNet is to allow the UE to access pico-cells that overlap within a geographical coverage area even when the UE is within the coverage of a donor macro-cell. In other words, when the traffic in the donor macro-cell increases, its UE can access the pico-cells automatically. Through this mechanism, HetNet can increase system capacity, particularly when there is the heavy traffic in the macro-cell.Relay systems are an alternative approach to denser infrastructures that are primarily useful in improving cell edge performance in macro-cells [17][18][19]. In this scheme, a relay node (RN) installed at a macro-cell edge where the received signal power from the serving eNB is low can transmit a strengthened signal to the UE. Thus, the RN can physically reduce the distance between eNB and UE. Three-dimensional (3D) beamforming is another key technology for achieving 5G in which antenna beams can be individually tailored to each UE in the elevation domain through the use of a massive multiple-input and multiple-output (MIMO) antenna architectures. The use of 3D beamforming by MIMO antennas enables directs signal transmission toward a target UE, potentially increasing the received signal-...