Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Xu, Zhi‐Qin John; Zhang, Yaoyu; Luo, Tao; Xiao, Yanyang; Ma, Zheng

doi:10.4208/cicp.oa-2020-0085

Cited by 217 publications

(134 citation statements)

References 19 publications

Supporting

Mentioning

124

Contrasting

Order By: Relevance

“…These works have signified the importance of the F-Principle. Theoretically, Xu et al [32] propose a theorem for the characterization of the initial training stage of a two-layer tanh network, which is also adopted in the analysis of DNNs with ReLU activation function [23]. Another series of works [4,6,24,34,36] attempt to understand the F-Principle in very wide neural networks, which can be well approximated by the first-order expansion with respect to the network parameters (the linear neural tangent kernel (NTK) regime).…”

Section: Introductionmentioning

confidence: 99%

“…In this work, we take another approach that uses Fourier analysis to study the learning behavior of DNNs based on the phenomenon of Frequency Principle (F-Principle), i.e., a DNN tends to learn a target function from low to high frequencies during the training [23,31,32,36]. Empirically, the F-Principle can be widely observed in general DNNs for both benchmark and synthetic data [31,32]. Conceptually, it provides a qualitative explanation of the success and failure of DNNs [32].…”

Section: Introductionmentioning

confidence: 99%

“…Empirically, the F-Principle can be widely observed in general DNNs for both benchmark and synthetic data [31,32]. Conceptually, it provides a qualitative explanation of the success and failure of DNNs [32]. E et al, (2019) [30] propose a continuous viewpoint for studying machine learning and suggest that the F-Principle underlying the gradient flows may be a main reason behind the success of modern machine learning.…”

Section: Introductionmentioning

confidence: 99%

“…Following the same direction as in [32], in this work, we propose a theoretical frame-work of Fourier analysis for the study of the training behavior of general DNNs in the following three stages: the initial stage, the intermediate stage, and the final stage. At all stages, we rigorously characterize the F-Principle by estimating some proper quantities.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Theory of the Frequency Principle for General Deep Neural Networks

Luo¹,

Ma²,

Xu³

et al. 2021

CSIAM-AM

View full text Add to dashboard Cite

Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, empirical studies reported a universal phenomenon of Frequency Principle (F-Principle), that is, a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very useful in providing both qualitative and quantitative understandings of DNNs. In this paper, we rigorously investigate the F-Principle for the training dynamics of a general DNN at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our results are general in the sense that they work for multilayer networks with general activation functions, population densities of data, and a large class of loss functions. Our work lays a theoretical foundation of the F-Principle for a better understanding of the training process of DNNs.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Theory of the Frequency Principle for General Deep Neural Networks

Luo¹,

Ma²,

Xu³

et al. 2021

CSIAM-AM

View full text Add to dashboard Cite

show abstract

“…Moreover, steganalysis pays more attention to the high-frequency texture information of the image, which is the position where the secret information embedding probability is higher in the adaptive steganography algorithm. According to the frequency principle [20], the deeper the network, the more likely it is to learn low-frequency image content information, which is inconsistent with the goal of steganalysis. Therefore, a too deep network limits the improvement of steganographic detection performance.…”

Section: Introductionmentioning

confidence: 99%

Multi-Frequency Residual Convolutional Neural Network for Steganalysis of Color Images

Lin¹,

2021

IEEE Access

View full text Add to dashboard Cite

Current steganalyzers based on deep learning mostly adopt wider or deeper designs to improve detection performance. However, an overly complex network increases the training cost and is not conducive to its expansion and optimization. Moreover, steganalysis pays more attention to high-frequency information corresponding to the image texture. However, the deeper the network, the more likely it is to learn lowfrequency information corresponding to the image content, which is inconsistent with the goal of steganalysis. In response to these problems, a multi-frequency residual deep convolutional neural network for steganalysis of color images called MFRNet is proposed in this paper. We apply the idea of multi-frequency analysis to steganography detection for the first time, effectively controlling the network scale. By designing columns of different depths, it can learn different frequency components of steganographic noise at the same time. The detection performance is better than the existing networks that only learn a single frequency component of steganographic noise at the same depth. Therefore, it can achieve a good detection performance with a lighter architecture. In addition, by designing residual basic blocks with different residual shortcuts, different scales of steganographic noise residuals can be calculated at the same time, which can effectively suppress the interference of image content, so as to better reduce the impact of steganography algorithm mismatch and payload mismatch than the existing methods. The experimental results on PPG-LIRMM-COLOR showed that the proposed MFRNet outperformed the state-of-the-art model WISERNet.

show abstract

Reliability analysis of heat source layout temperature field prediction considering uncertainty in deep neural network surrogate models

Xia

Yao

Zheng

et al. 2023

Quality & Reliability Eng

View full text Add to dashboard Cite

In engineering systems such as energy systems and electronic devices, thermal reliability analysis of the heat source component area is critical to ensure the proper functioning of these components. Previous methods, due to the high data dimensionality of the temperature field of the whole heat source area, face high computational cost when analyzing the reliability. To solve the problem of high computational cost, reliability analysis methods based on deep neural network surrogate models are receiving more and more attention. However, the uncertainty of deep neural networks is rarely considered in reliability problems, leading to unreasonable or imperfect reliability analysis. Therefore, we propose a reliability analysis method for heat source layout temperature field prediction, considering the uncertainty of deep neural network surrogate models. The method uses the information on the standard deviation of temperature field prediction provided by the model uncertainty to obtain the interval prediction of the temperature field. Then, the temperature field reliability interval of the layout area can be obtained by combining the temperature field interval prediction and Monte Carlo sampling. Further, we propose a strategy for selecting the heat source layout scheme based on the reliability interval in the late engineering design stage. Finally, the method's validity is verified using some examples, which provides an important reference basis for engineers.

show abstract

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Cited by 217 publications

References 19 publications

Theory of the Frequency Principle for General Deep Neural Networks

Theory of the Frequency Principle for General Deep Neural Networks

Multi-Frequency Residual Convolutional Neural Network for Steganalysis of Color Images

Reliability analysis of heat source layout temperature field prediction considering uncertainty in deep neural network surrogate models

Contact Info

Product

Resources

About