The efficiency of biometric systems, in particular efficient and accurate biometric identification, is one of the most challenging open problems in biometrics today. Adding to that, biometric data are sensitive data deserving adequate protection. As a solution, this work proposes an efficient privacy-preserving reduction of the computational workload of biometric identification systems using public-key encryption with keyword search (PEKS). For long-term protection of the biometric data, fully homomorphic encryption is applied for template protection. As all applied cryptographic schemes are lattice-based, they also offer post-quantum security. Throughout the system, the recognition accuracy of the unprotected system is preserved. In an evaluation on a public face database, the computational workload of an identification search in the encrypted domain is reduced down to 8.4% compared to an exhaustive search, achieving identification on 1062 subjects in 210 milliseconds. Based on these results, an identification search on 1 million subjects can be estimated at under 3 minutes using off-the-shelf hardware.