DPCL: Contrastive representation learning with differential privacy

Abstract

With the proliferation of unlabeled data, increasing efforts have been devoted to unsupervised learning. As one of the most representative branches of unsupervised learning, contrastive learning has made great progress with its high efficiency. Unfortunately, privacy threats to contrastive learning have become sophisticated, making it imperative to develop effective technologies that can deal with such threats. To alleviate the privacy issue in contrastive learning, we propose some novel techniques based on differential privacy, which aim at reducing the high sensitivity of gradient in the private training caused by interactive contrastive learning. Specifically, we add differentially private protection to the connection point related to different per-example gradients, which decreases the sensitivity of the gradients significantly. Our experiments on SimCLR and the Barlow Twins show that our approach is superior since it is more accurate while maintaining the same level of privacy protection.