Nowadays autonomous underwater vehicles (AUVs) are employed as unmanned machines in ocean industries. For instance AUVs play an important role in coastal area monitoring and investigating underwater pipe line in deep seas. In this paper, navigation of an AUV near free surface and the effect of wave disturbance and unmodeled hydrodynamics as uncertain terms in control system are addressed. To stabilize the roll motion of the vehicle a practical control mode in mini-UVs is applied. Firstly, a 6-DOF nonlinear dynamic simulator is developed and dynamic stability of the vehicle is investigated. Then, a feedback linearization method is applied to turn the nonlinear system into a convenient linear one, and then a robust technique is applied to guarantee the stability and performance of the system. In addition, a genetic algorithm method is employed for achieving the best gains in feedback linearization control law. Three constraints are considered for optimization including amplitude of sway, yaw and roll motions. Final results show an effective motion control of the AUV in horizontal plane. Meanwhile a reasonable performance of robust control in presence of wave disturbance and un-modeled hydrodynamics is achieved.