In the next generation wireless communication systems, high-speed data rate, reliable link quality, and ubiquitous access are necessary requirement. To meet the next generation communication system requirements, the ultra-dense small cell network (SCN) is proposed as one of the possible candidate solutions. To achieve high data rate and reliable link quality, coordinated multiple point (CoMP) transmission is usually used in ultra-dense SCN to satisfy the performance target. However, to realize CoMP transmissions in ultradense SCN, the feedback load is heavy because of the large amount of small cells and users. In this study, we want to investigate the ultra-dense SCN environment and design an effective method for its downlink (DL) transmission. This method consists of iterative scheme which iteratively solves the received signal with proper step-size value γ. The step-size value γ is very important to the convergence performance of iterative scheme because it affects the convergence speed and the converged error level of the iterative algorithm. Therefore, in this paper, we propose a novel machine learning based method which creates data model from input data. When the model is well established, the optimal step-size can be well estimated and the feedback information can become rough and the bandwidth allocated for feedback can be saved for data transmission. The simulations show that, the proposed method can create good model and achieve better convergence performance. For example, about the distribution of the transmissions with convergence iteration number less than 10 4 level, the proposed method can obtain 30% improvement than the traditional method with fixed step-size γ = 0.01. INDEX TERMS Small cell network (SCN), ultra-dense SCN, coordinated multi-point (CoMP) transmission, feedback reduction, machine learning, neural network.