Fourier neural networks as function approximators and differential equation solvers

Ngom, Marième; Marin, Oana

doi:10.1002/sam.11531

Cited by 13 publications

(5 citation statements)

References 16 publications

(31 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ref. [22] proposed trigonometric activation functions, which by coincidence are very similar to the example that will be presented below. The design and choice of these activations are, however, seldomly guided by principles and methods, but rather trial and error.…”

Section: Neural Networkmentioning

confidence: 81%

A Connection between Probability, Physics and Neural Networks

Ranftl

2022

The 41st International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering

View full text Add to dashboard Cite

show abstract

Section: Neural Networkmentioning

confidence: 81%

A Connection between Probability, Physics and Neural Networks

Ranftl

2022

The 41st International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering

View full text Add to dashboard Cite

show abstract

“…Periodic activation functions have seen limited adoption in machine learning outside of specific areas such as Fourier Neural Networks [20] and Implicit Neural Representations [21]. This is attributable primarily to the training difficulties introduced by the non-convex loss landscapes of neurons with periodic activations [22].…”

Section: Periodic Activation Functionsmentioning

confidence: 99%

Improving Vision Transformers with Novel GLU-Type Nonlinearities

Byrne,

Murray,

Marshall

2024

Preprint

View full text Add to dashboard Cite

In this study we systematically investigate a range of nonlinear functions in the fully-connected feed-forward portions of Vision Transformer (ViT) machine learning models. We limit our investigation to only GLU-type nonlinear functions, as the GLU-type function SwiGLU is the current standard in state-ofthe-art language and vision models. We identify 21 candidate layer functions with 1, 2, or 3 weight matrices, utilizing the activations: Sigmoid, Tanh, and Sin. ViT-Tiny models are implemented utilizing these functions and benchmarked on the image classification datasets CIFAR10, CIFAR100, and SVHN. Through these experiments we identify several previously uninvestigated functions which consistently outperform SwiGLU on all benchmarks. The most performant of these functions we call SinGLU. We further benchmark SwiGLU and SinGLU on the image classification dataset Imagenet64, and again SinGLU performs better. We note that periodic functions such as Sin are not common in neural networks. We perform a numerical investigation into sinusoidally activated neurons and suggest that their viability in Vision Transformers may be due to the losslandscape smoothing of Layer Normalization, Multi-Headed Self Attention (MHSA), and modern data augmentations such as label smoothing. However, our experiments on the CIFAR100 dataset indicate that layer nonlinearity remains a hyperparameter, with approximately piecewise-linear functions performing better than more complex functions when the problem space is not densely sampled.

show abstract

“…We here capitalize on the straightforward formalization offered by SysId methods. The AI alternatives form a pioneering field of research, which has recently garnered attention [34]: it leverages the algebraic similarities between the structure of conventional time regression methods and the convolution operations at the basis of convolutional neural networks. It is though still not clear whether such networks can be distilled to be ported on resource-constrained devices.…”

Section: Model Parameter Estimationmentioning

confidence: 99%

System Identification at the Extreme Edge for Network Load Reduction in Vibration-Based Monitoring

Zonzini

Dertimanis²,

Chatzi³

et al. 2022

IEEE Internet Things J.

View full text Add to dashboard Cite

Mechanical complexity, wide dimensions, and big data volume may hamper the implementation of Internet of Things (IoT)-enabled structural health monitoring (SHM) systems. In particular, one of the most important challenges is the reduction of the data payload to be transmitted over the monitoring network. Addressing the problem in the context of vibration-based SHM, this work explores system identification (SysId) as an innovative strategy for data compression at the extreme edge. Indeed, SysId is a signal processing technique aiming at finding a very reduced (i.e., less then one tenth of the total signal length) set of meaningful parameters, which can provide an alternative, but yet completely equivalent, frequency characterization of the structure. In the proposed approach, an embedded system-oriented adaptation of the sequential tall-skinny QR decomposition (eS-TSQR) from the dense linear algebra domain has been exploited to tackle both the memory and computational complexity of the involved algorithms. This yielded to the embodiment of input-output and output-only SysId models into a resource constrained device (i.e., an STM32L5 microcontoller unit), targeted on low-power and low-cost SHM applications, proving high effectiveness for the structural assessment of civil and industrial plants. Besides, a cost-benefit analysis is also presented, in which the energy saving brought by SysId running in a sensor-near manner is comprehensively measured against the power consumption due to data transmission, as implied by stateof-the-art communication protocols for IoT. Results demonstrate that SysId is 1.19× and 2.78× less energy demanding (with a payload reduction of 9× and 45×) w.r.t. compressed sensing-driven and compression-free solutions, respectively.

show abstract

Fourier neural networks as function approximators and differential equation solvers

Cited by 13 publications

References 16 publications

A Connection between Probability, Physics and Neural Networks

A Connection between Probability, Physics and Neural Networks

Improving Vision Transformers with Novel GLU-Type Nonlinearities

System Identification at the Extreme Edge for Network Load Reduction in Vibration-Based Monitoring

Contact Info

Product

Resources

About