Git Re-Basin: Merging Models modulo Permutation Symmetries

Ainsworth, Samuel; Hayase, Jonathan; Srinivasa, Siddhartha S.

doi:10.48550/arxiv.2209.04836

Cited by 8 publications

(15 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, compared to the most natural baseline of network alignment, DWSNets scale significantly better with the data. In reality, it is challenging to use this baseline due to the fact that the weight-space alignment problem is hard (Ainsworth et al, 2022). The problem is further amplified when having large input networks or large (networks) datasets.…”

Section: Analysis Of the Resultsmentioning

confidence: 99%

“…In a fundamental work, Hecht-Nielsen (1990) observed that MLPs have permutation symmetries: swapping the order of the activations in an intermediate layer does not change the underlying function. Motivated by previous works (Hecht-Nielsen, 1990;Brea et al, 2019;Ainsworth et al, 2022) we define the weight-space of an M -layer MLP as:…”

Section: Permutation Symmetries Of Neural Networkmentioning

confidence: 99%

“…The symmetries described in Equation (4) were used in several studies in the last few years, mainly to investigate the loss landscape of neural networks (Brea et al, 2019;Tatro et al, 2020;Arjevani & Field, 2021;Entezari et al, 2021;Simsek et al, 2021;Ainsworth et al, 2022;Peña et al, 2022), but also in Schürholt et al (2021) as a motivation for a data augmentation scheme.…”

Section: Permutation Symmetries Of Neural Networkmentioning

confidence: 99%

“…Therefore, from a symmetry and equivariance perspective, their work improperly models the symmetry group. Finally, in recent years several studies inspected the problem of aligning the weights of NNs (Ashmore & Gashler, 2015;Yurochkin et al, 2019;Wang et al, 2019;Singh & Jaggi, 2020;Tatro et al, 2020;Entezari et al, 2021;Ainsworth et al, 2022;Wang et al, 2022). As stated in the main text, solving the alignment tasks is hard and these strategies suffer from scaling issues to large datasets.…”

Section: Acknowledgementsmentioning

confidence: 99%

“…Approximation for model alignment. In the literature several studies suggested methods to align the weights/neurons of NNs (e.g., (Ashmore & Gashler, 2015;Singh & Jaggi, 2020;Ainsworth et al, 2022)). Here, we chose the method presented in (Ainsworth et al, 2022).…”

Section: Sine Wavesmentioning

confidence: 99%

See 4 more Smart Citations

Equivariant Architectures for Learning in Deep Weight Spaces

Navon¹,

Shamsian²,

Achituve³

et al. 2023

Preprint

View full text Add to dashboard Cite

Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domain to editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we present here a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trained MLP and processes it using a composition of layers that are equivariant to the natural permutation symmetry of the MLP's weights: Changing the order of neurons in intermediate layers of the MLP does not affect the function it represents. We provide a full characterization of all affine equivariant and invariant layers for these symmetries and show how these layers can be implemented using three basic operations: pooling, broadcasting, and fully connected layers applied to the input in an appropriate manner. We demonstrate the effectiveness of our architecture and its advantages over natural baselines in a variety of learning tasks.

show abstract

Section: Analysis Of the Resultsmentioning

confidence: 99%

Section: Permutation Symmetries Of Neural Networkmentioning

confidence: 99%

Section: Permutation Symmetries Of Neural Networkmentioning

confidence: 99%

Section: Acknowledgementsmentioning

confidence: 99%

Section: Sine Wavesmentioning

confidence: 99%

See 3 more Smart Citations

Equivariant Architectures for Learning in Deep Weight Spaces

Navon¹,

Shamsian²,

Achituve³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Model Fusion via Neuron Transplantation

Öz,

Kiefer,

Debus

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A Framework for Evaluating the Efficacy of Foundation Embedding Models in Healthcare

Xu,

Gui,

Rotemberg

et al. 2024

Preprint

View full text Add to dashboard Cite

Recent interest has surged in building large-scale foundation models for medical applications. In this paper, we propose a general framework for evaluating the efficacy of these foundation models in medicine, suggesting that they should be assessed across three dimensions: general performance, bias/fairness, and the influence of confounders. Utilizing Google's recently released dermatology embedding model and lesion diagnostics as examples, we demonstrate that: 1) dermatology foundation models surpass state-of-the-art classification accuracy; 2) general-purpose CLIP models encode features informative for medical applications and should be more broadly considered as a baseline; 3) skin tone is a key differentiator for performance, and the potential bias associated with it needs to be quantified, monitored, and communicated; and 4) image quality significantly impacts model performance, necessitating that evaluation results across different datasets control for this variable. Our findings provide a nuanced view of the utility and limitations of large-scale foundation models for medical AI.

show abstract

Git Re-Basin: Merging Models modulo Permutation Symmetries

Cited by 8 publications

References 0 publications

Equivariant Architectures for Learning in Deep Weight Spaces

Equivariant Architectures for Learning in Deep Weight Spaces

Model Fusion via Neuron Transplantation

A Framework for Evaluating the Efficacy of Foundation Embedding Models in Healthcare

Contact Info

Product

Resources

About