Relaxing Equivariance Constraints with Non-stationary Continuous Filters

Abstract

Equivariances provide useful inductive biases in neural network modeling, with the translation equivariance of convolutional neural networks being a canonical example. Equivariances can be embedded in architectures through weight-sharing and place symmetry constraints on the functions a neural network can represent. The type of symmetry is typically fixed and has to be chosen in advance. Although some tasks are inherently equivariant, many tasks do not strictly follow such symmetries. In such cases, equivariance constraints can be overly restrictive. In this work, we propose a parameter-efficient relaxation of equivariance that can effectively interpolate between a (i) non-equivariant linear product, (ii) a strict-equivariant convolution, and (iii) a strictly-invariant mapping. The proposed parameterization can be thought of as a building block to allow adjustable symmetry structure in neural networks. Compared to non-equivariant or strict-equivariant baselines, we experimentally verify that soft equivariance leads to improved performance in terms of test accuracy on CIFAR-10 and CIFAR-100 image classification tasks.

Data-driven Lie point symmetry detection for continuous dynamical systems

“…Symmetries in neural networks. Work has been done on studying group convolutional neural networks and using Lie groups to generalize beyond discrete and compact symmetry groups using localized kernels [7]. Although inspiring, this work is specialized to convolutions and kernel-based methods; in addition, the transformation groups to be sought are specified beforehand.…”

“…Symmetry detection, the task of discovering invariant properties from raw data, has been of increasing interest to the machine learning community [1][2][3][4], in particular for science and engineering applications [5,6]. For one, detecting symmetries allows for better interpretability of data [7] or even the sparse identification of underlying mechanisms [8]. Furthermore, incorporating symmetries into the learning algorithm improves robustness and trustworthiness [9] by making sure that predictions obey properties that the underlying mechanism obeys as well (e.g.…”

“…Third, the parametrization proposed in this paper alleviates the computational issue of expressing symmetry transformations for a much broader class of functions while retaining its predictive power for canonical symmetries. This was done by treating the transformations as differential operators [6,16,17] rather than the usual matrix group approach [1,2,7]. Overall, assuming the symmetries can be predicted accurately, the proposed algorithm allows for scaling up and simplifying the process of discovering symmetries.…”

Data Augmentation vs. Equivariant Networks: A Theory of Generalization on Dynamics Forecasting

“…A perfectly equivariant model may have trouble learning partial or approximated symmetries in real-world data. Thus, some work has recently explored the idea of building approximately equivariant models and empirically demonstrated the benefits of it in modeling real-world data (van der Ouderaa et al, 2022;Romero & Lohit, 2021;Finzi et al, 2021). For example, (Wang et al, 2022) designed approximately equivariant models by relaxing the weight sharing schemes in the equivariant convolution networks.…”

“…Several recent work have designed approximately equivariant networks (Wang et al, 2022;van der Ouderaa et al, 2022;Finzi et al, 2021) to learn the approximate functions.…”