In order to solve the problem of the poor recognition effect of traditional face recognition methods under pose, facial expression and illumination, a face recognition algorithm based on Multi-directional histograms of orthogonal oriented gradient is proposed. Firstly, the gradient characteristics of eight different directions, are extracted by using improved gradient operator. Secondly, counting the histogram feature of gradient direction in each direction and the Hog feature of eight directional orthogonal gradients, in order to get the overall image characteristics, is connected together. At last, overall image characteristics will be classified by nearest neighborhood classifier. The experimental result, which is tested in face library of Yale, ORL, CAS-PEAL-R1, shows that the proposed algorithm can improve the recognition rate effectively and has excellent robustness to the illumination and expression.