The learning-based hashing has recently made encouraging progress in face recognition. However, most existing hashing methods disregard the discrete constraint during optimization, inducing the accumulated quantization errors. In this work, we develop an effective learning-based hashing model, namely local feature hashing with binary auto-encoder (LFH-BAE), to directly learn local binary descriptors in the Hamming space. It attempts to exploit structure factors to well reconstruct the face image from binary codes. Specifically, we first introduce a binary auto-encoder to learn a hashing function to project each face region into high-quality binary codes. Since the original problem is a tricky combinational function, we then present a softened version to decompose it into separate tractable sub-problems. Next, we propose an effective alternating algorithm based on the augmented Lagrange method (ALM) to solve these sub-problems, which helps to generate strong discriminative and excellent robust binary codes. Moreover, we utilize the discrete cyclic coordinate descent (DCC) method to optimize binary codes to reduce the loss of useful information. Lastly, we cluster and pool the obtained binary codes, and construct a histogram feature as the final face representation for each image. Extensive experimental results on four public datasets including FERET, CAS-PEAL-R1, LFW and PaSC show that our LFH-BAE is superior to most state-of-the-art face recognition algorithms. INDEX TERMS Binary auto-encoder, binary feature, discrete optimization, face recognition, feature hashing.