Stable Network Morphism

Wei, Tao; Wang, Changhu; Chen, Chang Wen

doi:10.1109/ijcnn.2019.8851955

Cited by 53 publications

(80 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another approach to speed up performance estimation is to initialize the weights of novel architectures based on weights of other architectures that have been trained before. One way of achieving this, dubbed network morphisms [64], allows modifying an architecture while leaving the function represented by the network unchanged [10,11,21,22]. This allows increasing capacity of networks successively and retaining high performance without requiring training from scratch.…”

Section: Performance Estimation Strategymentioning

confidence: 99%

Neural Architecture Search

Elsken

Metzen

Hutter

2019

The Springer Series on Challenges in Machine Learning

1,237

1,077

View full text Add to dashboard Cite

Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation. One crucial aspect for this progress are novel neural architectures. Currently employed architectures have mostly been developed manually by human experts, which is a time-consuming and error-prone process. Because of this, there is growing interest in automated neural architecture search methods. We provide an overview of existing work in this field of research and categorize them according to three dimensions: search space, search strategy, and performance estimation strategy.

show abstract

Section: Performance Estimation Strategymentioning

confidence: 99%

Neural Architecture Search

Elsken

Metzen

Hutter

2019

The Springer Series on Challenges in Machine Learning

1,237

1,077

View full text Add to dashboard Cite

show abstract

“…Network morphism. In [4,32], a systematic study has been done on how to morph a well-trained neural network into a new one so that its network function can be completely preserved for further training. This network morphism can constitute a severe attack against our watermark because it may be impossible to detect the embedded watermark if the topology of the host network is severely modified.…”

Section: Future Workmentioning

confidence: 99%

Embedding Watermarks into Deep Neural Networks

Uchida

Nagai

Sakazawa

et al. 2017

Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval

475

570

View full text Add to dashboard Cite

Significant progress has been made with deep neural networks recently. Sharing trained models of deep neural networks has been a very important in the rapid progress of research and development of these systems. At the same time, it is necessary to protect the rights to shared trained models. To this end, we propose to use digital watermarking technology to protect intellectual property and detect intellectual property infringement in the use of trained models. First, we formulate a new problem: embedding watermarks into deep neural networks. We also define requirements, embedding situations, and attack types on watermarking in deep neural networks. Second, we propose a general framework for embedding a watermark in model parameters, using a parameter regularizer. Our approach does not impair the performance of networks into which a watermark is placed because the watermark is embedded while training the host network. Finally, we perform comprehensive experiments to reveal the potential of watermarking deep neural networks as the basis of this new research effort. We show that our framework can embed a watermark during the training of a deep neural network from scratch, and during fine-tuning and distilling, without impairing its performance. The embedded watermark does not disappear even after fine-tuning or parameter pruning; the watermark remains complete even after 65% of parameters are pruned.

show abstract

“…Network morphism. In [8,51], a systematic study has been conducted on how to morph a well-trained neural network into a new one so that its network function can be completely preserved for further training. This network morphism can constitute a severe attack against our watermark because it may be impossible to detect the embedded watermark if the topology of the host network undergoes major modification.…”

Section: Further Expected Developmentsmentioning

confidence: 99%

Digital watermarking for deep neural networks

Nagai

Uchida²,

Sakazawa

et al. 2018

Int J Multimed Info Retr

124

View full text Add to dashboard Cite

Although deep neural networks have made tremendous progress in the area of multimedia representation, training neural models requires a large amount of data and time. It is well-known that utilizing trained models as initial weights often achieves lower training error than neural networks that are not pre-trained. A fine-tuning step helps to reduce both the computational cost and improve performance. Therefore, sharing trained models has been very important for the rapid progress of research and development. In addition, trained models could be important assets for the owner(s) who trained them, hence we regard trained models as intellectual property. In this paper, we propose a digital watermarking technology for ownership authorization of deep neural networks. First, we formulate a new problem: embedding watermarks into deep neural networks. We also define requirements, embedding situations, and attack types on watermarking in deep neural networks. Second, we propose a general framework for embedding a watermark in model parameters, using a parameter regularizer. Our ap- S. Satoh National Institute of Informatics 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, 101-8430, Japan E-mail: satoh@nii.ac.jp proach does not impair the performance of networks into which a watermark is placed because the watermark is embedded while training the host network. Finally, we perform comprehensive experiments to reveal the potential of watermarking deep neural networks as the basis of this new research effort. We show that our framework can embed a watermark during the training of a deep neural network from scratch, and during fine-tuning and distilling, without impairing its performance. The embedded watermark does not disappear even after fine-tuning or parameter pruning; the watermark remains complete even after 65% of parameters are pruned.

show abstract

Stable Network Morphism

Cited by 53 publications

References 28 publications

Neural Architecture Search

Neural Architecture Search

Embedding Watermarks into Deep Neural Networks

Digital watermarking for deep neural networks

Contact Info

Product

Resources

About