The presence of inertia perturbation should not be ignored, it may harm the stability of quadrotor. There are many causes, one of them is due to the payload attached to the quadrotor. Here, the inertia perturbation comes from the payload that subject to the laying position and the mass. Although unknown, the mass and position of payload are limited to certain value. In this paper, we provide the robust Proportional-Derivative (PD) control of a quadrotor while considering the inertia perturbation. The control law is designed by generating torques in SO(3) group which will regulate the angles and the angular velocity to reach the equilibrium points. In the same time, the inertia perturbation is countered by a proposed compensator. The effectiveness of the control law is clarified by using numerical simulation. The results show that the control law prevails in stabilizing the quadrotor in both nominal condition and the condition under inertia perturbation. It is verified that the proposed control law can stabilize the quadrotor despite being under inertia perturbation.