Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations

Abstract

Contrastive learning is a highly effective method which uses unlabeled data to produce representations which are linearly separable for downstream classification tasks. Recent works have shown that contrastive representations are not only useful when data come from a single domain, but are also effective for transferring across domains. Concretely, when contrastive representations are trained on data from two domains (a source and target) and a linear classification head is trained to predict labels using only the labeled source data, the resulting classifier also exhibits good transfer to the target domain. In this work, we analyze this linear transferability phenomenon, building upon the framework proposed by HaoChen et al. ( 2021) which relates contrastive learning to spectral clustering of a positive-pair graph on the data. We prove that contrastive representations capture relationships between subpopulations in the positive-pair graph: linear transferability can occur when data from the same class in different domains (e.g., photo dogs and cartoon dogs) are connected in the graph. Our analysis allows the source and target classes to have unbounded density ratios and be mapped to distant representations. Our proof is also built upon technical improvements over the main results of HaoChen et al. (2021), which may be of independent interest.

Understanding Representation Learnability of Nonlinear Self-Supervised Learning

“…Theoretical analyses for final data representation. For the analysis of the data representation learnability, many works focus on the final data representation (the optimal solution of the pretext task) and measure the quality of the final data representation in the downstream tasks by using a linear classifier (HaoChen et al 2021(HaoChen et al , 2022Arora et al 2019;Lee et al 2021;Tosh, Krishnamurthy, and Hsu 2021). The main difference in this line of work is how to obtain the final data representation.…”

“…Despite the empirical success of SSL (He et al 2020;Chen et al 2020;Chen and He 2021;Zhong et al 2022), there are only a few works that focus on data representation learnability (Arora et al 2019;Tosh, Krishnamurthy, and Hsu 2021;Lee et al 2021;HaoChen et al 2021HaoChen et al , 2022Tian 2022a,b;Wen and Li 2021;Liu et al 2021). However, studying the learnability is helpful in understanding why SSL models can obtain meaningful data representations.…”

“…Arora et al (2019) obtained the data representation function by minimizing the empirical SSL loss in a special data representation function class. HaoChen et al (2021) and HaoChen et al (2022) studied final data representation by closed-form solutions. They viewed the nonlinear neural network as a "black box" and ignored the learning result of the nonlinear neural network.…”