A multi-degree-of-freedom Permanent Magnet Spherical Actuator (PMSpA) has a special mechanical structure and electromagnetic fields, and is easily affected by nonlinear disturbances such as modeling errors and friction. Therefore, the quality of a PMSpA control system may be deteriorated. In order to keep the PMSpA with good trajectory tracking performance, this paper designs a time delay estimation controller based on gradient compensation. Firstly, the dynamic model of the PMSpA with nonlinear terms is derived. The nonlinear terms in the complex dynamic model can be simplified and estimated by the time delay estimation method. Secondly, for the estimation errors caused by time delay control, a gradient compensator is introduced to further correct and compensate for it. Furthermore, the stability of the designed controller is proved by the Lyapunov equation. Finally, the correctness and effectiveness of the controller are validated by comparison with other controllers through simulation. In addition, experimental results have also shown that the control accuracy of the spherical motor can be effectively improved using the proposed controller.