This paper proposes a control scheme combining improved particle swarm optimization (IPSO) and adaptive linear active disturbance rejection control (ALADRC) to solve the high-speed train (HST) speed tracking control problem. Firstly, in order to meet the actual operation of a HST, a multi-mass point dynamic model with time-varying coefficients was established. Secondly, linear active disturbance rejection control (LADRC) was proposed to control the speed of the HST, and the anti-disturbance ability of the system was improved by estimating and compensating for the total disturbance suffered by the carriage during the operation of the HST. Meanwhile, to solve the problem of difficult parameter tuning of the LADRC, IPSO was introduced to optimize the parameters. Thirdly, the adaptive control (APC) was introduced to compensate for the observation error caused by the bandwidth limitation of the linear state expansion observer in LADRC and the tracking error caused by an unknown disturbance during the train’s operation. Additionally, the Lyapunov theory was used to prove the stability of the system. Finally, the simulation results showed that the designed control scheme is more effective in solving the problem of HST speed tracking.