Information-Theoretic Generalization Bounds for Meta-Learning and Applications

Jose, Sharu Theresa; Simeone, Osvaldo

doi:10.3390/e23010126

Cited by 21 publications

(16 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, the distribution of the topologies is maximally diverse at some intermediate value of the interference radius. In line with this observation, metalearning is seen to profit from task diversity, which prevents meta-overfitting [21].…”

Section: Resultsmentioning

confidence: 57%

Fast Power Control Adaptation via Meta-Learning for Random Edge Graph Neural Networks

Nikoloska

Simeone

2021

2021 IEEE 22nd International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)

Self Cite

View full text Add to dashboard Cite

Power control in decentralized wireless networks poses a complex stochastic optimization problem when formulated as the maximization of the average sum rate for arbitrary interference graphs. Recent work has introduced data-driven design methods that leverage graph neural network (GNN) to efficiently parametrize the power control policy mapping channel state information (CSI) to the power vector. The specific GNN architecture, known as random edge GNN (REGNN), defines a non-linear graph convolutional architecture whose spatial weights are tied to the channel coefficients, enabling a direct adaption to channel conditions. This paper studies the higher-level problem of enabling fast adaption of the power control policy to time-varying topologies. To this end, we apply first-order meta-learning on data from multiple topologies with the aim of optimizing for a few-shot adaptation to new network configurations.

show abstract

Section: Resultsmentioning

confidence: 57%

Fast Power Control Adaptation via Meta-Learning for Random Edge Graph Neural Networks

Nikoloska

Simeone

2021

2021 IEEE 22nd International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this setting, we directly analyze the effects of generic, domain-invariant, and targeted augmentations on OOD risk. Our analysis is related to work on metalearning that considers generalization to a meta-distribution (Chen et al, 2021a;Jose & Simeone, 2021); however, these analyses focus on adaptation to new tasks instead of out-of-domain generalization.…”

Section: Related Workmentioning

confidence: 99%

Out-of-Domain Robustness via Targeted Augmentations

Gao¹,

Sagawa²,

Koh³

et al. 2023

Preprint

View full text Add to dashboard Cite

Models trained on one set of domains often suffer performance drops on unseen domains, e.g., when wildlife monitoring models are deployed in new camera locations. In this work, we study principles for designing data augmentations for out-of-domain (OOD) generalization.In particular, we focus on real-world scenarios in which some domain-dependent features are robust, i.e., some features that vary across domains are predictive OOD. For example, in the wildlife monitoring application above, image backgrounds vary across camera locations but indicate habitat type, which helps predict the species of photographed animals. Motivated by theoretical analysis on a linear setting, we propose targeted augmentations, which selectively randomize spurious domain-dependent features while preserving robust ones. We prove that targeted augmentations improve OOD performance, allowing models to generalize better with fewer domains. In contrast, existing approaches such as generic augmentations, which fail to randomize domain-dependent features, and domain-invariant augmentations, which randomize all domain-dependent features, both perform poorly OOD. In experiments on three realworld datasets, we show that targeted augmentations set new states-of-the-art for OOD performance by 3.2-15.2%.

show abstract

“…Additionally, (Denevi et al, 2019; studied algorithms which incrementally update the bias regularization parameter using a sequence of observed tasks. Another line of research is studying the meta-generalization gap, and finding bounds on it on average (Jose & Simeone, 2021;Rezazadeh et al, 2021) or with high probability (Pentina & Lampert, 2014;Amit & Meir, 2018;Rothfuss et al, 2021;Liu et al, 2021;Guan et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

A General framework for PAC-Bayes Bounds for Meta-Learning

Rezazadeh¹

2022

Preprint

View full text Add to dashboard Cite

Meta learning automatically infers an inductive bias, that includes the hyperparameter of the base-learning algorithm, by observing data from a finite number of related tasks. This paper studies PAC-Bayes bounds on meta generalization gap. The meta-generalization gap comprises two sources of generalization gaps: the environmentlevel and task-level gaps resulting from observation of a finite number of tasks and data samples per task, respectively. In this paper, by upper bounding arbitrary convex functions, which link the expected and empirical losses at the environment and also per-task levels, we obtain new PAC-Bayes bounds. Using these bounds, we develop new PAC-Bayes meta-learning algorithms. Numerical examples demonstrate the merits of the proposed novel bounds and algorithm in comparison to prior PAC-Bayes bounds for meta-learning.

show abstract

Information-Theoretic Generalization Bounds for Meta-Learning and Applications

Cited by 21 publications

References 31 publications

Fast Power Control Adaptation via Meta-Learning for Random Edge Graph Neural Networks

Fast Power Control Adaptation via Meta-Learning for Random Edge Graph Neural Networks

Out-of-Domain Robustness via Targeted Augmentations

A General framework for PAC-Bayes Bounds for Meta-Learning

Contact Info

Product

Resources

About