This paper studies trajectory tracking control problem of quadrotor in the presence of external disturbances. A new control scheme integrating adaptive sliding mode disturbance observer (ASMDO) with backstepping funnel control is designed in both position loop and attitude loop. Adaptive sliding mode disturbance observer is utilized to estimate and compensate the external disturbances of quadrotor system, which relaxed the upper bound requirement in conventional sliding mode disturbance observer (SMDO) whilst adapting the gain of discontinuous term to a level as small as possible for consideration of chattering mitigation. The modified funnel function not only guarantees the dynamic quality in the tracking process but also avoids derivative operation of Euclidian norm of tracking error in the backstepping design. To guarantee stabilization of closed-loop system, Lyapunov stability criterion is utilized in the overall controller design process. Finally, the effectiveness and superiority of the proposed scheme are illustrated by simulation results.