As the evolution of trends of intelligent IoT in 5G era, ultra-dense networks (UDN) become a promising paradigm via densely deploying small cells in cellular networks, where the transmission rate of mobile users can be highly improved. In this paper, an investigative study was presented regarding optimizing deployment of small cell base stations (BS) to maximize the average sum rate (ASR) in 5G UDN. In particular on a stochastic geometrical perspective, a homogeneous-type Poisson point process (PPP) was used for depicting an arbitrary arrangement of both macro cell user equipment (UE) and small cell BSs. Moreover, the closed-form probabilities of successful transmission was derived regarding the uplink and downlink of small cells. Then, the ASR of small cells was obtained as well as the problem of maximizing ASR was analyzed with outage constraints. Further, the study also demonstrated that the maximum ASR located in a closed interval of small cell BS density, where the lower and higher bounds of the interval were obtained. Finally, for maximizing the ASR value, the optimal small cell BS density in a closed-form was derived out with convex optimization theory. Simulation analysis indicated that different constraints from the macro cell network led to different maximum ASRs, and interferences caused by small cells and macro cell UE had likewise influenced the performance of small cells.