While the widespread application of biometric identification technology has provided convenient services, the security, and privacy of personal data have also attracted increasing attention. Because biometric traits are unique, once an attack occurs, the information stored in the system will be permanently leaked. Thus, ensuring the security of biometric templates is a major challenge. In order to solve this problem, we propose a finger vein template protection method based on chaotic map. First, we randomize and scramble the finger vein features after feature extraction, disturbing the location distribution of the pixels. Second, on the basis of the chaotic features of the finger veins, binary quantization is applied to further ensure the safety of the template. Finally, the compact and safe finger vein template is directly stored in the database. Finger vein matching is realized in the encryption domain, which changes the way that the traditional finger vein recognition system directly stores biometric templates. We conducted a series of detailed experiments on the public dataset. The experimental results show that our method can ensure the safety of finger vein templates while maintaining stable recognition performance. Moreover, the method has universality for different feature-extraction methods and meets the requirements of good biometric template protection.