This paper presents a cascade-modified uncertainty and disturbance estimator (CMUDE)–based controller. The attitude-tracking performance of a quadrotor would be greatly degraded due to two factors: (1) a nonideal actuator and (2) an exogenous disturbance. Based on these factors, an actuator model is accurately identified using a thrust test platform. Then, a cascade-structured CMUDE controller is designed through the backstepping method, so the disturbances from different channels can be separately dealt with. The idea of the uncertainty and disturbance estimator (UDE) is then applied to dynamically compensate for not only the exogenous disturbance but also the time delay in the actuator dynamics. Benefitting from that, the problem of mismatched disturbances is solved by defining virtual control inputs; furthermore, the nonlinear term in the attitude kinematics is included in the model uncertainty to be canceled instead of being ignored. When tracking attitude commands, the exogenous disturbance and model uncertainty along with the time delay of the actuator are estimated and compensated for by the proposed controller, so the performance of the disturbance rejection is significantly improved while maintaining a quick response to the rapidly changing commands. To verify that, the proposed controller is implemented on a quadrotor platform to carry out exhaustive experiments and compared with the mainstream cascade PID (CPID) controller, the classic UDE-based controller, and the latest modified-UDE (MUDE)–based controller. The results show that our control strategy achieves a better disturbance rejection and more accurate tracking performance than other controllers.