Massive machine-type communications (mMTCs) for Internet of things are being developed thanks to the fifth-generation (5G) wireless systems. Narrowband Internet of things (NB-IoT) is an important communication technology for machine-type communications. It supports many different protocols for communication. The reliability and performance of application layer communication protocols are greatly affected by the retransmission time-out (RTO) algorithm. In order to improve the reliability and performance of machine-type communications, this study proposes a novel RTO algorithm UDP-XGB based on the user datagram protocol (UDP) and NB-IoT. It combines traditional algorithms with machine learning. The simulation results show that real round-trip time (RTT) is close to the RTO, which is obtained by this algorithm, and the reliability and performance of machine-type communications have improved.