To solve the non-unique problem of inverse solutions of nonlinear periodic function and the uncertainties in the transmission mechanism of vibration displacement system of continuous casting mold driven by servo motor, an adaptive high-order sliding mode control(HOSMC) strategy based on nonlinear processing method is proposed. Firstly, a new nonlinear processing method is utilized to solve the problem that the output equation of the system has the non-unique problem of inverse solutions of a nonlinear periodic function, which is convenient for the controller design of displacement loop. Secondly, an adaptive HOSMC controller is proposed to suppress the uncertainties including the initial zero position deviation of the eccentric shaft, parameter perturbation, and fast time-varying load disturbance, and track the desired displacement accurately in the displacement loop system, and it is proved that the displacement tracking error is stable and convergent to the near origin in finite time. Finally, the simulation and experimental researches are all verified that the proposed control method is superior to the PI control method in the field and has better dynamic and steady-state tracking performance.