This paper investigates the impact of mutual coupling (MC) on the achievable sum-rate of power-domain non-orthogonal multiple access (NOMA) system in both uplink (UL) and downlink (DL) transmissions. We assume a single-antenna user NOMA system with directional beamforming in the millimeter wave (mmW) channel. Due to the electromagnetic interaction between the antenna elements, called mutual coupling, at the base station (BS), the steering vector of the channel is affected. Consequently, this leads to distorted antenna pattern pointing towards different directions and reduction in channel gain, based on antenna array structure. In this paper, different antenna geometries and configurations, such as onedimensional (1D) uniform linear array (ULA), two-dimensional (2D), and three-dimensional (3D) uniform circular array (UCA), are implemented at the BS, where induced electromotive force (EMF) and method of moment (MOM) techniques are used to generate the MC coefficients. We first examine the antenna patterns affected by MC, followed by channel gains of user terminals (UT)s. Then sum-rate of the NOMA system is modified in the presence of MC. Furthermore, mutual coupling is compensated successfully for both UL and DL systems for the given antenna structures. Simulation results show that mutual coupling degrades the sum-rate performance of the NOMA system in all three array structures, especially in the UCA structure due to the smaller spacing of the array element in a circular shape, resulting in symmetric mutual coupling from both sides of the circle. On the other hand, it is also shown that compensating the mutual coupling effect by the MOM technique in the case of unknown MC or matrix inversion in the case of known MC significantly improves the system sum-rate in all scenarios. INDEX TERMS Non-orthogonal multiple access (NOMA), multiuser beamforming, millimeter wave, sum-rate, mutual coupling (MC), uniform linear array (ULA), uniform circular array (UCA), 3D UCA.