Biometric systems store sensitive personal data that need to be highly protected. However, state-of-the-art template protection schemes generally consist of separate processes, inspired by salting, hashing, or encryption, that limit the achievable performance. Moreover, these are inadequate to protect current state-of-the-art biometric models as they rely on end-to-end deep learning methods. After proposing the Secure Triplet Loss, focused on template cancelability, we now reformulate it to address the problem of template linkability. Evaluated on biometric verification with off-the-person electrocardiogram (ECG) and unconstrained face images, the proposed method proves successful in training secure biometric models from scratch and adapting a pretrained model to make it secure. The results show that this new formulation of the Secure Triplet Loss succeeds in optimizing end-to-end deep biometric models to verify template cancelability, non-linkability, and non-invertibility.