As a cellular network of the fifth generation (5G) is commercialized, mobile devices and data throughput have rapidly increased. According to the spatial density for communication increases, the nodes of cell are overloaded. Therefore, the heterogeneous ultra dense network (UDN) is suggested. Furthermore, the techniques for selecting a relay have been proposed in the Heterogeneous Net (HetNet). The relays are needed to improve communication performance and mitigate overload of nodes. In this paper, an adaptive relay selection scheme is proposed to obtain the diversity gain from multiple relays. To enhance the reliability of communication, the proposed scheme suggests a new algorithm considering outage probability and diversity gain of compound channel. Furthermore, the selected relays use an antenna selection algorithm to improve the channel capacity. Simulation results show that the proposed scheme improves the bit error rate (BER) and the data throughput.