With the rapid increase in wind power, its strong randomness has brought great challenges to power system operation. Accurate and timely ultra-short-term wind power prediction is essential for the stable operation of power systems. In this paper, an LsAdam–LSTM model is proposed for ultra-short-term wind power prediction, which is obtained by accelerating the long short-term memory (LSTM) network using an improved Adam optimizer with loss shrinkage (LsAdam). For a specific network topology, training progress heavily depends on the learning rate. To make the training loss of LSTM shrink faster with standard Adam, we use the past training loss-changing information to finely tune the next learning rate. Therefore, we design a gain coefficient according to the loss change to adjust the global learning rate in every epoch. In this way, the loss change in the training process can be incorporated into the learning progress and a closed-loop adaptive learning rate tuning mechanism can be constructed. Drastic changes in network parameters will deteriorate learning progress and even make the model non-converging, so the gain coefficient is designed based on the arctangent function with self-limiting properties. Because the learning rate is iteratively tuned with past loss-changing information, the trained model will have better performance. The test results on a wind turbine show that the LsAdam–LSTM model can obtain higher prediction accuracy with much fewer training epochs compared with Adam–LSTM, and the prediction accuracy has significant improvements compared with BP and SVR models.