Data Symmetries and Learning in Fully Connected Neural Networks

Abstract

Symmetries in the data and how they constrain the learned weights of modern deep networks is still an open problem. In this work we study the simple case of fully connected shallow non-linear neural networks and consider two types of symmetries: full dataset symmetries where the dataset X is mapped into itself by any transformation g, i.e. gX = X or single data point symmetries where gx = x, x ∈ X . We prove and experimentally confirm that symmetries in the data are directly inherited at the level of the network's learned weights and relate these findings with the common practice of data augmentation in modern machine learning. Finally, we show how symmetry constraints have a profound impact on the spectrum of the learned weights, an aspect of the so-called network implicit bias.

Synergy between AI and Optical Metasurfaces: A Critical Overview of Recent Advances

“…One must be careful when performing such classification, because there are DNNs that do not belong solely to a single group. For instance, Fully Connected Neural Networks (FCNNs) [270], also known as dense networks, can be applied to all four groups, because these DNNs are adaptable and thus can be utilized in different manners, as determined by a specific goal. A similar point of view is generally valid for stochastic neural networks [271], since one can introduce stochasticity into NNs belonging to any of the four main groups.…”