A tracking control algorithm based on active disturbance rejection controller (ADRC) is proposed to overcome the telescope’s mount fluctuation. The fluctuations are caused by internal and external disturbance when the large aperture telescope runs at ultra-low speed with direct drive. According to the high-precision and high-stability requirements of a large aperture telescope, the ADRC position controller is designed based on segmented arc Permanent Magnet Synchronous Motors (arc PMSMs). The tracking differentiator of ADRC is designed to undergo a transition process to avoid overshoot in the position loop. The speed of target tracking process is observed by the extended state observer and the position information in the system is estimated in real time. The current control variable of the segmented arc PMSM is generated by implementing a non-linear state error feedback control law. The simulation results demonstrate that the proposed control strategy can not only accurately estimate the position and speed of the tracking target, but also estimate the disturbance to compensate the control variables. Experiments showed that the speed error is less than 0.05′ s−1 when using the ADRC, and it can realize high tracking performance when compared with PID controller, which improves the robustness of a large aperture telescope control system.