Ultra-dense networks (UDNs) have been employed to solve the pressing problems in relation to the increasing demand for higher coverage and capacity of the fifth generation (5G) wireless networks. The deployment of UDNs in a very large scale has been envisioned to break the fundamental deadlocks of beyond 5G or the sixth generation (6G) networks and deliver many more orders of magnitude gains that today's technologies achieve. However, the mathematical tool to optimize the system performance under the stringent radio resource constraints is widely recognized to be a formidable challenge. System-level performance optimization of current UDNs are usually conducted by relying on numerical simulations, which are often time-consuming and have become extremely difficult in the context of 6G with extremely high density. As such, there is an urgent need for developing a realistic mathematical model for optimizing the 6G UDNs. In this paper, we introduce challenges as well as issues that have to be thoroughly considered while deploying UDNs in realistic environment. We revisit efficient mathematical techniques including game theory and real-time optimization in the context of optimizing UDNs performance. In addition, emerging technologies which are suitable to apply in UDNs are also discussed. Some of them have already been used in UDNs with high efficiency while the others which are still under investigation are expected to boost the performance of UDNs to achieve the requirements of 6G. Importantly, for the first time, we introduce the joint optimal approach between realtime optimization and game theory (ROG) which is an effective tool to solve the optimization problems of large-scale UDNs with low complexity. Then, we describe two approaches for using ROG in UDNs. Finally, some case study of ROG are given to illustrate how to apply ROG for solving the problems of different applications in UDNs.