HIDRA: Head Initialization across Dynamic targets for Robust Architectures

Abstract

The performance of gradient-based optimization strategies depends heavily on the initial weights of the parametric model. Recent works show that there exist weight initializations from which optimization procedures can find the taskspecific parameters faster than from uniformly random initializations, and that such a weight initialization can be learned by optimizing a specific model architecture across similar tasks via maml (Model-Agnostic Meta-Learning). Current methods are limited to populations of classification tasks that share the same number of classes due to the static model architectures used during meta-learning. In this paper, we present hidra, a meta-learning approach that enables training and evaluating across tasks with any number of target variables. We show that Model-Agnostic Meta-Learning trains a distribution for all the neurons in the output layer and a specific weight initialization for the ones in the hidden layers. Hydra explores this by learning one master neuron which is used to initialize any number of output neurons for a new task. Extensive experiments on the Miniimagenet and Omniglot data sets demonstrate that hidra improves over standard approaches while generalizing to tasks with any number of target variables. Moreover, our approach is shown to robustify low-capacity models in learning across complex tasks with a high number of classes for which regular MAML fails to learn any feasible initialization.

Few-Shot Forecasting of Time-Series with Heterogeneous Channels

“…One of the first methods to attempt few-shot learning on homogeneous predictors was chameleon [4] which used a convolutional encoder to align tasks from similar domains to a common attribute space before utilizing gradient-based few-shot methods. Similarly, other works tried to learn across tasks with a varied label spaces [9,32]. Finally, Iwata et al [19] proposed a model that uses deep set [53] based blocks to compute a task-embedding over predictor and targets of training samples (support data) which then can be combined with new unlabeled samples (query data) to perform a classification or regression without the need of retraining or fine-tuning, similar to neighbor-based approaches (we will refer to this method as HetNet throughout the rest of the paper).…”