A novel sliding mode controller (SMC) with nonlinear fractional order PID sliding surface based on a novel extended state observer for the speed operation of a surface-mounted permanent magnet synchronous motor (SPMSM) is proposed in this paper. First, a new smooth and derivable nonlinear function with improved continuity and derivative is designed to replace the traditional nonderivable nonlinear function of the nonlinear state error feedback control law. Then, a nonlinear fractional order PID sliding mode controller is proposed on the basis of the fractional order PID sliding surface with the combination of the novel nonlinear state error feedback control law to improve dynamic performance, static performance, and robustness of the system. Furthermore, a novel extended state observer is designed based on the new nonlinear function to achieve dynamic feedback compensation for external disturbances. Stability of the system is proved based on the Lyapunov stability theorem. The corresponding comparative simulation results demonstrate that the proposed composite control algorithm displays good stability, dynamic properties, and strong robustness against external disturbances.