Supermasks in Superposition

Wortsman, Mitchell; Ramanujan, Vivek; Liu, Rosanne; Kembhavi, Aniruddha; Rastegari, Mohammad; Yosinski, Jason; Farhadi, Ali

doi:10.48550/arxiv.2006.14769

Cited by 10 publications

(23 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Transfer and cross-domain learning Transfer learning as a field is quite varied. Methods are variously classified under few-shot [24,44], continual learning [34,47], and lifelong learning [27,38]. In general, transfer learning methods seek to use knowledge learned from one domain in another to improve performance [33].…”

Section: Related Workmentioning

confidence: 99%

Forward Compatible Training for Large-Scale Embedding Retrieval Systems

Ramanujan¹,

Vasu²,

Farhadi³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In visual retrieval systems, updating the embedding model requires recomputing features for every piece of data. This expensive process is referred to as backfilling. Recently, the idea of backward compatible training (BCT) was proposed. To avoid the cost of backfilling, BCT modifies training of the new model to make its representations compatible with those of the old model. However, BCT can significantly hinder the performance of the new model. In this work, we propose a new learning paradigm for representation learning: forward compatible training (FCT). In FCT, when the old model is trained, we also prepare for a future unknown version of the model. We propose learning sideinformation, an auxiliary feature for each sample which facilitates future updates of the model. To develop a powerful and flexible framework for model compatibility, we combine side-information with a forward transformation from old to new embeddings. Training of the new model is not modified, hence, its accuracy is not degraded. We demonstrate significant retrieval accuracy improvement compared to BCT for various datasets: ImageNet-1k (+18.1%), Places-365 (+5.4%), and VGG-Face2 (+8.3%). FCT obtains model compatibility when the new and old models are trained across different datasets, losses, and architectures.

show abstract

Section: Related Workmentioning

confidence: 99%

Forward Compatible Training for Large-Scale Embedding Retrieval Systems

Ramanujan¹,

Vasu²,

Farhadi³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Parametric isolation based methods adaptively introduce new parameters for new tasks to avoid the parameters of previous tasks being drastically changed [32], [33], [34], [35], [36]. For instance, progressive network [32] allocates a new sub-network for each new task and block any modification on the previously learned networks.…”

Section: Continual Learningmentioning

confidence: 99%

“…Yoon et al [33] proposed a more flexible model (DEN) that dynamically adds new neurons to accommodate new tasks. Recently, various innovative approaches to allocate separated parameters for different tasks have been developed [34], [35], [36]. Besides the concrete models, Knoblauch et al [37] analyzed the required capability of an optimal continual learning agent.…”

Section: Continual Learningmentioning

confidence: 99%

“…In this way, the number of examples contained in different classes of each task are arranged to be as balanced as possible. Specifically, the class indices of each task are: { (35,12), (15,21), (28,30), (16,24), (10,34), (8,4), (5,2), (27,26), (36,19), (23,31), (9,37), (13,3), (20,39), (22,6), (38,33), (25,11), (18,1), (14,7), (0, 17), (29,32)}, where each tuple denotes a task consisting of two classes.…”

Section: Datasetsmentioning

confidence: 99%

“…In our experiments, we select 46 classes and omit the last class containing only 1 example. Similar to OGB-Arxiv, we reorder the classes and the class indices of each tasks are: {(4, 7), (6, 3), (12, 2), (0, 8), (1,13), (16,21), (9,10), (18,24), (17,5), (11,42), (15,20), (19,23), (14,25), (28,29), (43,22), (36,44), (26,37), (32,31), (30,27), (34,38), (41,35), (39,33), (45,40)}.…”

Section: Actor Co-occurrence Networkmentioning

confidence: 99%

See 2 more Smart Citations

Hierarchical Prototype Networks for Continual Graph Representation Learning

Zhang¹,

Song²,

Tao³

2021

Preprint

View full text Add to dashboard Cite

Despite significant advances in graph representation learning, little attention has been paid to the more practical continual learning scenario in which new categories of nodes (e.g., new research areas in citation networks, or new types of products in co-purchasing networks) and their associated edges are continuously emerging, causing catastrophic forgetting on previous categories. Existing methods either ignore the rich topological information or sacrifice plasticity for stability. To this end, we present Hierarchical Prototype Networks (HPNs) which extract different levels of abstract knowledge in the form of prototypes to represent the continuously expanded graphs. Specifically, we first leverage a set of Atomic Feature Extractors (AFEs) to encode both the elemental attribute information and the topological structure of the target node. Next, we develop HPNs to adaptively select relevant AFEs and represent each node with three levels of prototypes. In this way, whenever a new category of nodes is given, only the relevant AFEs and prototypes at each level will be activated and refined, while others remain uninterrupted to maintain the performance over existing nodes. Theoretically, we first demonstrate that the memory consumption of HPNs is bounded regardless of how many tasks are encountered. Then, we prove that under mild constraints, learning new tasks will not alter the prototypes matched to previous data, thereby eliminating the forgetting problem. The theoretical results are supported by experiments on five datasets, showing that HPNs not only outperform state-of-the-art baseline techniques but also consume relatively less memory.

show abstract

Aquaculture systems in Hanoi, Vietnam.

Nguyen¹

2020

Preprint

View full text Add to dashboard Cite

Hanoi, the capital of Vietnam, is one of the most densely populated areas in the world, with about 3,000 people km-2. Traditionally, aquaculture is practiced in ponds, reservoirs and urban lakes, rice fields, wastewater and VAC [an acronym from the Vietnamese words for garden (vuon), pond (ao) and livestock quarters (chuong)] systems. With over 5,000 ha of surface area and 3,000 ha of lowland rice fields, Hanoi has potential for further aquaculture development and tourism-entertainment services. The importance of aquaculture has long been recognized as a source of income and employment. Nowadays, pond aquaculture in peri-urban areas is decreasing due to urbanization, while in areas far from the city it is increasing through the conversion of lowland rice fields to fish ponds.

show abstract

Supermasks in Superposition

Cited by 10 publications

References 29 publications

Forward Compatible Training for Large-Scale Embedding Retrieval Systems

Forward Compatible Training for Large-Scale Embedding Retrieval Systems

Hierarchical Prototype Networks for Continual Graph Representation Learning

Aquaculture systems in Hanoi, Vietnam.

Contact Info

Product

Resources

About