Quadrotor drone attitude tracking is inevitably affected by the combination of model uncertainties, external disturbances, and non-ideal actuator dynamics during stable flight and complex maneuvers. In order to achieve precise attitude control in these situations, a cascade-structured dynamic surface control (DSC) strategy is proposed based on an uncertainty and disturbance estimator (UDE), considering the actuator dynamics as represented by a first-order plus time-delay model. The DSC scheme is used to transform the original attitude dynamics system into a set of interconnected subsystems. On the one hand, the mismatched disturbances in the attitude kinematics and dynamics loops are converted into matched disturbances to accommodate the structural constraints of the UDE so that these disturbances, as well as the non-ideality caused by the actuator time delay, are estimated and compensated for by the approach. On the other hand, the “complexity explosion” problem is addressed by the first-order filter employed by DSC. The ultimate boundedness of the closed-loop system is proven while the parameter design constraints are provided. MATLAB Simulink simulations are conducted to demonstrate the desirability of considering actuator dynamics and to verify that the proposed control strategy can relax the constraints of the control parameters and enable a higher accuracy.