On Universal Features for High-Dimensional Learning and Inference

Huang, Shao-Lun; Makur, Anuran; Wornell, Gregory W.; Zheng, Lizhong

doi:10.48550/arxiv.1911.09105

Cited by 6 publications

(22 citation statements)

References 89 publications

(206 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here, H-score is a metric determining whether certain features are informative for the task (more formally introduced by Bao et al 2019) 1 . Its derivation is rooted in the maximal correlation interpretation of deep neural networks (Huang et al 2019a), where given the extracted features, the formula of H-score is actually the optimal network performance under an information theoretic measurement. Previous work also extended H-score to a transferability metric and verified its effectiveness with extensive experiments (Bao et al 2019;Ibrahim, Ponomareva, and Mazumder 2022), suggesting the potential of H-score in transfer learning.…”

Section: Related Work Maximal Correlation Regression and H-scorementioning

confidence: 99%

H-ensemble: An Information Theoretic Approach to Reliable Few-Shot Multi-Source-Free Transfer

Wu,

Wang,

Wang

et al. 2024

AAAI

View full text Add to dashboard Cite

Multi-source transfer learning is an effective solution to data scarcity by utilizing multiple source tasks for the learning of the target task. However, access to source data and model details is limited in the era of commercial models, giving rise to the setting of multi-source-free (MSF) transfer learning that aims to leverage source domain knowledge without such access. As a newly defined problem paradigm, MSF transfer learning remains largely underexplored and not clearly formulated. In this work, we adopt an information theoretic perspective on it and propose a framework named H-ensemble, which dynamically learns the optimal linear combination, or ensemble, of source models for the target task, using a generalization of maximal correlation regression. The ensemble weights are optimized by maximizing an information theoretic metric for transferability. Compared to previous works, H-ensemble is characterized by: 1) its adaptability to a novel and realistic MSF setting for few-shot target tasks, 2) theoretical reliability, 3) a lightweight structure easy to interpret and adapt. Our method is empirically validated by ablation studies, along with extensive comparative analysis with other task ensemble and transfer learning methods. We show that the H-ensemble can successfully learn the optimal task ensemble, as well as outperform prior arts.

show abstract

Section: Related Work Maximal Correlation Regression and H-scorementioning

confidence: 99%

H-ensemble: An Information Theoretic Approach to Reliable Few-Shot Multi-Source-Free Transfer

Wu,

Wang,

Wang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…− p(x)p(y), for discrete random variables x and y. It is shown in [27] that, under the assumption of sufficiently small B, the solution of the cross-entropy loss coincides with the following solution of the matrix decomposition:…”

Section: H-scorementioning

confidence: 99%

“…However, the key drawback of the H-score is that the optimality is based on the least squares objective, which is rarely used for classification. As proven in [27], the least squares solution is a valid approximation to the cross-entropy classification loss only when label y and input x are weakly dependent, which is clearly not the case in general.…”

Section: H-scorementioning

confidence: 99%

“…Furthermore, the predictor and the metric are estimated on the same dataset, which is prone to overfitting. On the other hand, the least squares solution of H-score is generally not a valid approximation to the commonly-used cross-entropy loss for classification, unless the dependence between the label and the input feature is weak [27], which rarely holds in practice.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks

Ding¹,

Chen²,

Levinboim³

et al. 2022

Preprint

View full text Add to dashboard Cite

With the increasing abundance of pretrained models in recent years, the problem of selecting the best pretrained checkpoint for a particular downstream classification task has been gaining increased attention. Although several methods have recently been proposed to tackle the selection problem (e.g. LEEP, H-score), these methods resort to applying heuristics that are not well motivated by learning theory. In this paper we present PACTran, a theoretically grounded family of metrics for pretrained model selection and transferability measurement. We first show how to derive PACTran metrics from the optimal PAC-Bayesian bound under the transfer learning setting. We then empirically evaluate three metric instantiations of PACTran on a number of vision tasks (VTAB) as well as a language-and-vision (OKVQA) task. An analysis of the results shows PACTran is a more consistent and effective transferability measure compared to existing selection methods.

show abstract

“…The advantage of using bivariate distributions is that we can have probabilistic insights on network analysis. A very interesting recent work on the embedding problem for a bipartite network [35] also used a bivariate distribution to characterize a useritem network. There they showed that the optimal embedding vectors can be interpreted as conditional expectations.…”

Section: Equivalence Of the Embedding Problem In Sampled Graphsmentioning

confidence: 99%

Explainable, Stable, and Scalable Graph Convolutional Networks for Learning Graph Representation

Lu,

Chang

2020

Preprint

View full text Add to dashboard Cite

The network embedding problem that maps nodes in a graph to vectors in Euclidean space can be very useful for addressing several important tasks on a graph. Recently, graph neural networks (GNNs) have been proposed for solving such a problem. However, most embedding algorithms and GNNs are difficult to interpret and do not scale well to handle millions of nodes. In this paper, we tackle the problem from a new perspective based on the equivalence of three constrained optimization problems: the network embedding problem, the trace maximization problem of the modularity matrix in a sampled graph, and the matrix factorization problem of the modularity matrix in a sampled graph. The optimal solutions to these three problems are the dominant eigenvectors of the modularity matrix. We proposed two algorithms that belong to a special class of graph convolutional networks (GCNs) for solving these problems: (i) Clustering As Feature Embedding GCN (CAFE-GCN) and (ii) sphere-GCN. Both algorithms are stable trace maximization algorithms, and they yield good approximations of dominant eigenvectors. Moreover, there are linear-time implementations for sparse graphs. In addition to solving the network embedding problem, both proposed GCNs are capable of performing dimensionality reduction. Various experiments are conducted to evaluate our proposed GCNs and show that our proposed GCNs outperform almost all the baseline methods. Moreover, CAFE-GCN could be benefited from the labeled data and have tremendous improvements in various performance metrics.

show abstract

On Universal Features for High-Dimensional Learning and Inference

Cited by 6 publications

References 89 publications

H-ensemble: An Information Theoretic Approach to Reliable Few-Shot Multi-Source-Free Transfer

H-ensemble: An Information Theoretic Approach to Reliable Few-Shot Multi-Source-Free Transfer

PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks

Explainable, Stable, and Scalable Graph Convolutional Networks for Learning Graph Representation

Contact Info

Product

Resources

About