In this paper, a robust dual-channel disturbance rejection control based on an inner-outer loop control framework is proposed for the attitude control of the quadrotor under modeling uncertainties and unknown disturbances. In the outer loop, a tracking differentiator is introduced to obtain smooth tracking signals and their derivatives. In addition, an outer loop controller is developed with the tracking signals set as the desired signals. In the inner loop, a robust dual-channel disturbance rejection controller based on a sliding mode observer is constructed, which contains an inner disturbance rejection channel (IDRC) and an outer disturbance rejection channel (ODRC). In the IDRC, a disturbance rejection compensator is designed to obtain the disturbance compensation values and to compensate a part of the lumped disturbances in the attitude dynamic model. In the ODRC, an inner loop controller with variable-gain switching terms and constant-gain switching terms is designed, whose switching terms are used to compensate the remaining part of the lumped disturbances in the attitude dynamic model. By using the proposed control scheme, the robustness is guaranteed, and the chattering phenomenon caused by the variable-gain switching terms is greatly reduced. The stability of the proposed scheme is analyzed by using the Lyapunov theory. Finally, the effectiveness is tested by the platform experiments.