In this paper, we propose a geometric control law for attitude robust control problem of a rigid body system without angular velocity measurement. We suppose there is a disturbance related to angular velocity and attitude in the system. The estimated values of attitude and angular velocity obtained by an angular velocity observer can converge to the true values exponentially by choosing appropriate control law. Furthermore, by designing an adaptive law for the inertia matrix, the exponential stabilization of the system can still be guaranteed without the knowledge of the true inertia matrix. Finally, the proposed theoretical results are verified by our simulation examples.