Prediction of landslide evolution state is important for early warning system of landslides. The displacement curve of reservoir landslides has step-like characteristics. However, the mutation point of displacement curve is difficult to predict. An optimized machine learning model based on Extreme Gradient Boosting (XGBoost) and Bayesian method (Baye-XGB) is proposed to predict mutation points of displacement curve. The accuracy of models was testified by the Baishuihe landslide. Rainfall, reservoir water level and former displacement are taken as input parameters. K-means cluster was used to classify mutation points and regular points. XGBoost is used to predict evolution state, and the Bayesian method is applied to search hyperparameters. The results indicate that Baye-XGB is better than other models such as Support vector machine (SVM) and artificial neural network (ANN). The monthly displacement greater than 50mm is classified as a mutation point, the monthly displacement smaller than 50mm is classified as a regular point. The F1-score and AUC of the Baye-XGB are 0.95 and 0.99, respectively. The AUC score of Baye-XGB is improved by 17.86% compared with XGBoost_NoSmote, which means the SMOTE disposition can greatly improve the accuracy. Therefore, Baye-XGB can provide scientific guidelines for landslide earning waring.