Non-orthogonal multiple access (NOMA) is a promising candidate beyond 5G and 6G to improve spectral efficiency by sharing the same resource block with other users. However, as the number of users increases in a single NOMA cluster, not only the spectral efficiency but also the interference increases. Therefore, to ensure a trade-off between capacity and error performance in NOMA systems, this study introduces a novel downlink NOMA scheme using the phase rotation. In the system, four users can create a single NOMA cluster and are grouped into two subgroups by exploiting conventional NOMA. Within each subgroup, the symbols of two users, near user and far user, are rotated and multiplexed by the power domain. Subsequently, the final transmitted signal is obtained by multiplexing the in-phase and quadrature components of the first and second subgroups, respectively. By superposing signals from two independent subgroups, the number of successive interference cancellation (SIC) operations are reduced compared to the existing NOMA. Moreover, the analytical bit error rate (BER) for a four-user scenario is derived, and the optimal rotation angle and power allocation are proposed to minimize the BER performance. Numerical results demonstrate the improved performance of the proposed NOMA scheme over several existing schemes, such as conventional NOMA and orthogonal multiple access (OMA).