Scaffolding Sets

Burhanpurkar, Maya; Deng, Zhun; Dwork, Cynthia; Zhang, Linjun

doi:10.48550/arxiv.2111.03135

Cited by 4 publications

(8 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Self-Supervised Contrastive Learning Another closely related line of research is unimodal self-supervised contrastive learning (SSCL) for unimodal data. Representation learning has been crucial in modern machine learning Bengio et al (2013); Burhanpurkar et al (2021); Zhang et al (2021); Deng et al (2021); Kawaguchi et al (2022). SSCL is a group of self-supervised learning algorithms that learn representations by contrasting two views generated by data augmentation.…”

Section: Related Workmentioning

confidence: 99%

Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data

Nakada¹,

Gulluk²,

Deng³

et al. 2023

Preprint

View full text Add to dashboard Cite

Language-supervised vision models have recently attracted great attention in computer vision. A common approach to build such models is to use contrastive learning on paired data across the two modalities, as exemplified by Contrastive Language-Image Pre-Training (CLIP). In this paper, under linear representation settings, (i) we initiate the investigation of a general class of nonlinear loss functions for multimodal contrastive learning (MMCL) including CLIP loss and show its connection to singular value decomposition (SVD). Namely, we show that each step of loss minimization by gradient descent can be seen as performing SVD on a contrastive cross-covariance matrix. Based on this insight, (ii) we analyze the performance of MMCL. We quantitatively show that the feature learning ability of MMCL can be better than that of unimodal contrastive learning applied to each modality even under the presence of wrongly matched pairs. This characterizes the robustness of MMCL to noisy data. Furthermore, when we have access to additional unpaired data, (iii) we propose a new MMCL loss that incorporates additional unpaired datasets. We show that the algorithm can detect the ground-truth pairs and improve performance by fully exploiting unpaired datasets. The performance of the proposed algorithm was verified by numerical experiments.

show abstract

Section: Related Workmentioning

confidence: 99%

Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data

Nakada¹,

Gulluk²,

Deng³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Roughly concurrently, work on 'multi-calibration' has suggested explicitly taking into account input information for selecting groups: The original paper on multi-calibration (Hébert-Johnson et al, 2018) proposed to look at all computable groups -a proposal that has also been made in the sequence setting by (Dawid, 1985). But even approaches based on similarity of learned representations either prescribe that the selected groups have equal predictions (Burhanpurkar et al, 2021) or apply additional binning in prediction space (Luo et al, 2022). 5 Approaches proposing to omit this additional binning are sometimes referred to as 'multi-accuracy' (Hébert-Johnson et al, 2018;Kim, Ghorbani, and Zou, 2019) but we are not aware of a thorough discussion of this choice.…”

Section: Conventional Understanding Of Calibrationmentioning

confidence: 99%

“…One alternative, suggested in (Burhanpurkar et al, 2021), would be to consider all efficiently computable groups.…”

Section: Grouping Choicesmentioning

confidence: 99%

“…This can be desirable, for example, in the context of fairness where we are interested in the performance of p on groups that are often in some way local or similar in the input space. Some recent approaches propose to group points based on similarity of representations learned by some feature extractor (Burhanpurkar et al, 2021;Luo et al, 2022) An advantage of such approaches is that they implicitly make use of both inputs and predictions while a disadvantage is that they do so in an opaque way. Especially for fairness applications, this might be difficult to justify since the ethical significance of such groups is less clear compared to groups defined explicitly in input or prediction space.…”

Section: Vicinity-based Grouping Principlesmentioning

confidence: 99%

See 1 more Smart Citation

On the Richness of Calibration

Höltgen¹,

Williamson²

2023

Preprint

View full text Add to dashboard Cite

Probabilistic predictions can be evaluated through comparisons with observed label frequencies, that is, through the lens of calibration. Recent scholarship on algorithmic fairness has started to look at a growing variety of calibration-based objectives under the name of multi-calibration but has still remained fairly restricted. In this paper, we explore and analyse forms of evaluation through calibration by making explicit the choices involved in designing calibration scores. We organise these into three grouping choices and a choice concerning the agglomeration of group errors. This provides a framework for comparing previously proposed calibration scores and helps to formulate novel ones with desirable mathematical properties. In particular, we explore the possibility of grouping datapoints based on their input features rather than on predictions and formally demonstrate advantages of such approaches. We also characterise the space of suitable agglomeration functions for group errors, generalising previously proposed calibration scores. Complementary to such population-level scores, we explore calibration scores at the individual level and analyse their relationship to choices of grouping. We draw on these insights to introduce and axiomatise fairness deviation measures for population-level scores. We demonstrate that with appropriate choices of grouping, these novel global fairness scores can provide notions of (sub-)group or individual fairness.

show abstract

“…Beyond fairness, multicalibration and OI also provide strong accuracy guarantees [see e.g. , Blum and Lykouris, 2020, Zhao, Kim, Sahoo, Ma, and Ermon, 2021, Gopalan, Kalai, Reingold, Sharan, and Wieder, 2022, Kim, Kern, Goldwasser, Kreuter, and Reingold, 2022, Burhanpurkar, Deng, Dwork, and Zhang, 2021. For a general predictor class P and a subpopulation class C, Shabat, Cohen, and Mansour [2020] showed sample complexity upper bounds of uniform convergence for multicalibration based on the maximum of suitable complexity measures of C and P. They complemented this result with a lower bound which does not grow with C and P. In comparison, we focus on the weaker no-access OI setting where the sample complexity can be much smaller, and we provide matching upper and lower bounds in terms of the dependence on D and P.…”

Section: Related Workmentioning

confidence: 99%

Metric Entropy Duality and the Sample Complexity of Outcome Indistinguishability

Hu¹,

Peale²,

Reingold³

2022

Preprint

View full text Add to dashboard Cite

We give the first sample complexity characterizations for outcome indistinguishability, a theoretical framework of machine learning recently introduced by Dwork, Kim, Reingold, Rothblum, and Yona (STOC 2021). In outcome indistinguishability, the goal of the learner is to output a predictor that cannot be distinguished from the target predictor by a class D of distinguishers examining the outcomes generated according to the predictors' predictions. While outcome indistinguishability originated from the algorithmic fairness literature, it provides a flexible objective for machine learning even when fairness is not a consideration. In this work, we view outcome indistinguishability as a relaxation of PAC learning that allows us to achieve meaningful performance guarantees under data constraint.In the distribution-specific and realizable setting where the learner is given the data distribution together with a predictor class P containing the target predictor, we show that the sample complexity of outcome indistinguishability is characterized by the metric entropy of P w.r.t. the dual Minkowski norm defined by D, and equivalently by the metric entropy of D w.r.t. the dual Minkowski norm defined by P. This equivalence makes an intriguing connection to the long-standing metric entropy duality conjecture in convex geometry. Our sample complexity characterization implies a variant of metric entropy duality, which we show is nearly tight. In the distribution-free setting, we focus on the case considered by Dwork et al. where P contains all possible predictors, hence the sample complexity only depends on D. In this setting, we show that the sample complexity of outcome indistinguishability is characterized by the fat-shattering dimension of D.We also show a strong sample complexity separation between realizable and agnostic outcome indistinguishability in both the distribution-free and the distribution-specific settings. This is in contrast to distribution-free (resp. distribution-specific) PAC learning where the sample complexity in both the realizable and the agnostic settings can be characterized by the VC dimension (resp. metric entropy).

show abstract

Scaffolding Sets

Cited by 4 publications

References 38 publications

Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data

Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data

On the Richness of Calibration

Metric Entropy Duality and the Sample Complexity of Outcome Indistinguishability

Contact Info

Product

Resources

About