Digital watermarking for deep neural networks

Abstract

Although deep neural networks have made tremendous progress in the area of multimedia representation, training neural models requires a large amount of data and time. It is well-known that utilizing trained models as initial weights often achieves lower training error than neural networks that are not pre-trained. A fine-tuning step helps to reduce both the computational cost and improve performance. Therefore, sharing trained models has been very important for the rapid progress of research and development. In addition, trained models could be important assets for the owner(s) who trained them, hence we regard trained models as intellectual property. In this paper, we propose a digital watermarking technology for ownership authorization of deep neural networks. First, we formulate a new problem: embedding watermarks into deep neural networks. We also define requirements, embedding situations, and attack types on watermarking in deep neural networks. Second, we propose a general framework for embedding a watermark in model parameters, using a parameter regularizer. Our ap- S. Satoh National Institute of Informatics 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, 101-8430, Japan E-mail: satoh@nii.ac.jp proach does not impair the performance of networks into which a watermark is placed because the watermark is embedded while training the host network. Finally, we perform comprehensive experiments to reveal the potential of watermarking deep neural networks as the basis of this new research effort. We show that our framework can embed a watermark during the training of a deep neural network from scratch, and during fine-tuning and distilling, without impairing its performance. The embedded watermark does not disappear even after fine-tuning or parameter pruning; the watermark remains complete even after 65% of parameters are pruned.

Adversarial frontier stitching for remote neural network watermarking

“…After extension to surprising domains such as network science [40], the extension to watermarking neural networks as objects themselves is new, following the need to protect the valuable assets of today's state of the art machine learning techniques. Uchida et al [35,22] thus propose the watermarking of neural networks, by embedding information in the learned weights. Authors show in the case of convolutional architectures that this embedding does not significantly change the distribution of parameters in the model.…”

Stealing Knowledge from Protected Deep Neural Networks Using Composite Unlabeled Data

“…Algorithm 1 provides the complete composite datageneration method, which is run at the beginning of each epoch. Figure 1 is an illustration of composite data samples created by Algorithm 1. for i=1 to N do 11: i 1 = math.random(len(dataArr)) 12: i 2 = math.random(len(dataArr)) 13: p = math.random(100)/100 14:…”

TamperNN: Efficient Tampering Detection of Deployed Neural Nets

“…Watermarking techniques [19,23] embed information into the target model weights in order to mark its provenance. Since work in [19] operated on the MNIST dataset, and provided detailed parameters, we implemented this watermarking technique on the same models (MLP, CNN, and IRNN).…”

“…Yet, very recently, other types of attacks were shown to operate by modifying the model itself, by embedding information in the model weight matrices. This is the case of new watermarking techniques, that aim at embedding watermarks into the model, in order to prove model ownership [19,23], or of trojaning attacks [4,21] that empowers the attacker to trigger specific model behaviors. Figure 1: Illustration of a model tampering and its impact on the decision boundaries.…”