This article investigates the handoff optimization in an uplink non-orthogonal multiple access-based small cell network (SCN) with successive interference cancellation that takes both the base station (BS) association and power control into consideration. Due to user mobility and varying communication environment, it is important to associate a user with the proper BS and control its transmit power by considering the user mobility for the SCNs. To overcome overly frequent re-association and handoffs between BSs, a novel joint optimization of handoff, BS association, and total transmit power is presented that maximizes the system-wide and handoff utilities and minimizes power consumption simultaneously. We define the handoff utility function by considering the BS association in each time frame with the previous time frame.Then, we formulate two inter-related optimization problems under the minimum average-rate and maximum allowable transmit power of mobile users (MUs) constraints. The designed optimization problems are combined through the weighted sum method into a single-stage optimization problem. The transformed problem is solved by adopting the coalition formation game and primal decomposition theories. We design the coalition value function by collectively considering the handoff number of each BS in every time frame, transmit power of all MUs, and the number of associated MUs to every BS. The simulation results show the efficiency of the proposed handoff algorithm in comparison to another BS association algorithm in terms of the number of handoffs.

The co‐channel interference and frequent handoff are two important issues that have a great impact on the spectrum and energy efficiency in the non‐orthogonal multiple access (NOMA) with successive interference cancellation (SIC) in the small cell networks (SCN). This paper investigates the performance of SIC decoding order based on the users’ channel state information (CSI) and random SIC decoding order under the perfect and imperfect conditions for the handoff optimization in SCNs with NOMA. The effect of SIC decoding order in the joint base station association and power control problem is studied by considering the motion of mobile users (MUs). Two optimization problems are formulated using the number of associated MUs, the number of handoffs, and the transmit power of all MUs, under the maximum allowable transmit power and minimum average‐rate constraints. Two objectives are combined into a single‐stage optimization problem through the weighted sum method. The formulated problem is solved using a game theory‐based algorithm and primal decomposition theory. The simulation results show that changing the SIC decoding order can lead to performance improvement in the uplink NOMA‐ based SCNs. Also, it is shown that the proposed algorithm can significantly reduce the frequent handoffs in small cell networks.