SSLGuard: A Watermarking Scheme for Self-supervised Learning Pre-trained Encoders

Abstract

Self-supervised learning is an emerging machine learning (ML) paradigm. Compared to supervised learning that leverages high-quality labeled datasets to achieve good performance, self-supervised learning relies on unlabeled datasets to pre-train powerful encoders which can then be treated as feature extractors for various downstream tasks. The huge amount of data and computational resources consumption makes the encoders themselves become a valuable intellectual property of the model owner. Recent research has shown that the ML model's copyright is threatened by model stealing attacks, which aims to train a surrogate model to mimic the behavior of a given model. We empirically show that pre-trained encoders are highly vulnerable to model stealing attacks. However, most of the current efforts of copyright protection algorithms such as fingerprinting and watermarking concentrate on classifiers. Meanwhile, the intrinsic challenges of pre-trained encoder's copyright protection remain largely unstudied. We fill the gap by proposing SSL-Guard, the first watermarking algorithm for pre-trained encoders. Given a clean pre-trained encoder, SSLGuard embeds a watermark into it and outputs a watermarked version. The shadow training technique is also applied to preserve the watermark under potential model stealing attacks. Our extensive evaluation shows that SSLGuard is effective in watermark injection and verification, and is robust against model stealing and other watermark removal attacks such as pruning and finetuning.

SSL-WM: A Black-Box Watermarking Approach for Encoders Pre-trained by Self-Supervised Learning

“…Recently, several backdoor injection and watermarking approaches have been proposed in the self-supervised learning domain. SSLGuard [12] proposes a watermarking approach to protect the IP of SSL pretrained encoders in computer vision tasks. It injects a secret keytuple into the encoders as the watermark and extracts the key from the output of the suspect encoder to verify the ownership by comparing the cosine similarity between the extracted key and the injected key.…”

“…For instance, StolenEncoder [41] aims to utilize model extraction to steal the encoder in a scenario where a single encoder is provided as API services without the downstream classifiers. Therefore, most existing watermark approaches [31], [12] just verify the ownership of encoders against model extraction attacks without the downstream tasks. SSLGuard [12] aims to verify the ownership of encoders against model extraction attacks where encoders are provided as the API services, but it may not work when a downstream task's classifier is connected after the encoder.…”

“…Therefore, most existing watermark approaches [31], [12] just verify the ownership of encoders against model extraction attacks without the downstream tasks. SSLGuard [12] aims to verify the ownership of encoders against model extraction attacks where encoders are provided as the API services, but it may not work when a downstream task's classifier is connected after the encoder. For example, SSLGuard fails to verify the ownership in CIFAR-10, CINIC-10, and GTSRB downstream tasks, as shown in Section V-E. Also, it is unknown if SSLGuard is robust against model extraction attacks in downstream tasks scenario.…”

“…Unfortunately, existing watermarking approaches cannot effectively verify the ownership of encoders in this scenario. SSLGuard [12] focuses on a scenario where the encoders are deployed separately without downstream classifiers. However, it cannot verify the ownership if the adversaries steal the encoder and transfer it to downstream tasks, since the stolen encoders and their outputs are typically not accessible from those downstream models.…”

AWEncoder: Adversarial Watermarking Pre-Trained Encoders in Contrastive Learning

“…The zero-bit watermark is often embedded into the model by enforcing the model to learn the mapping relationship between the carefully crafted samples and the pre-determined labels. However, with the rise of contrastive learning, the pretrained encoders are treated as feature extractors for various downstream tasks, and the pre-training process is relying on the SSL strategy rather than label-based supervised learning [26,27]. It indicates that traditional black-box watermarking techniques are not suitable for the pre-trained encoders in contrastive learning.…”