In target tracking, the tracking process needs to constantly update the data information. For maneuvering target, model mismatch and loss of high-order moment information disrupt the accuracy of the state estimation. In this paper, an adaptive high-order unscented Kalman filter (AHUKF) algorithm is proposed for the case of errors occurring in the capturing of model dynamic behavior using the classical unscented Kalman filter (UKF) algorithm. By introducing the free parameter, the analytical solution of the high-order unscented transformation (UT) was obtained, the basis for choosing free parameters was analyzed, and the stability of the algorithm was discussed. A method for obtaining the optimal adaptive factor based on the prediction residual estimation covariance matrix was proposed, which reduces the influence of the dynamic model error and was applied to the target tracking model. In this paper, the proposed AHUKF is applied to a target tracking problem with state mutation, different sampling intervals, and different turn rates, respectively. Simulation results for target tracking illustrate that the proposed algorithm is more accurate and robust than the UKF, high-order unscented Kalman filter (HUKF) and adaptive unscented Kalman filter (AUKF). INDEX TERMS UKF, optimal adaptive factor, adaptive filtering, orthogonality principle, high-order UT sampling.