Multicalibrated Partitions for Importance Weights

Gopalan, Parikshit; Reingold, Omer; Sharan, Vatsal; Wieder, Udi

doi:10.48550/arxiv.2103.05853

Cited by 3 publications

(21 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We show that the upper bound in Equation ( 13) only requires multiaccuracy (as opposed to full multi-group attribution). In contrast, [GRSW21] showed that neither direction of Equation ( 12) is implied by multiaccuracy alone.…”

Section: Sandwiching Bounds From Multi-group Attributionmentioning

confidence: 86%

“…The first condition is known as multiaccuracy [GRSW21], and there are many known algorithms that achieve it. It suffices to correctly estimate the marginal term.…”

Section: Multi-group Attributionmentioning

confidence: 99%

“…Since the current definition which requires exact equalities is not possible to achieve with finitely many samples from R and P , 2 we define the notion of (α, β) multigroup attribution which allows for some slack in both equations. Our main technical contribution is to show that multi-group attribution can be derived from the multi-calibrated partitions introduced by [GRSW21]. Informally, this is a partition of the domain X into regions S 1 , .…”

Section: Efficient Multi-group Attributionmentioning

confidence: 99%

“…Formally, our partitions satisfy a relaxed notion of this condition, which allows them to be computed efficiently. Given such a partition, our main result, Theorem 4.1 shows that the distribution Q = w • P where w(S i ) = R(S i )/P (S i ) for all x ∈ S i satisfies multi-group attribution for C. Combined with the algorithm from [GRSW21], we derive an efficient algorithm for multi-group attribution, assuming that C has a weak agnostic learner [BLM01,HKRR18,GRSW21].…”

Section: Efficient Multi-group Attributionmentioning

confidence: 99%

“…The work of [GRSW21] introduced sandwiching bounds as a multi-group fairness/accuracy desideratum for importance weights. The setting is that the learnt importance weights w are meant to approximate the true importance weights w * of the distribution R relative to P .…”

Section: Sandwiching Bounds From Multi-group Attributionmentioning

confidence: 99%

See 4 more Smart Citations

KL Divergence Estimation with Multi-group Attribution

Gopalan¹,

Narodytska²,

Reingold³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Estimating the Kullback-Leibler (KL) divergence between two distributions given samples from them is well-studied in machine learning and information theory. Motivated by considerations of multi-group fairness, we seek KL divergence estimates that accurately reflect the contributions of sub-populations to the overall divergence. We model the sub-populations coming from a rich (possibly infinite) family C of overlapping subsets of the domain. We propose the notion of multi-group attribution for C, which requires that the estimated divergence conditioned on every sub-population in C satisfies some natural accuracy and fairness desiderata, such as ensuring that sub-populations where the model predicts significant divergence do diverge significantly in the two distributions. Our main technical contribution is to show that multi-group attribution can be derived from the recently introduced notion of multi-calibration for importance weights [HKRR18, GRSW21]. We provide experimental evidence to support our theoretical results, and show that multi-group attribution provides better KL divergence estimates when conditioned on sub-populations than other popular algorithms.

show abstract

Section: Sandwiching Bounds From Multi-group Attributionmentioning

confidence: 86%

“…The first condition is known as multiaccuracy [GRSW21], and there are many known algorithms that achieve it. It suffices to correctly estimate the marginal term.…”

Section: Multi-group Attributionmentioning

confidence: 99%

Section: Efficient Multi-group Attributionmentioning

confidence: 99%

Section: Efficient Multi-group Attributionmentioning

confidence: 99%

Section: Sandwiching Bounds From Multi-group Attributionmentioning

confidence: 99%

See 3 more Smart Citations

KL Divergence Estimation with Multi-group Attribution

Gopalan¹,

Narodytska²,

Reingold³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Omnipredictors

Gopalan,

Kalai,

Reingold

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Loss minimization is a dominant paradigm in machine learning, where a predictor is trained to minimize some loss function that depends on an uncertain event (e.g., "will it rain tomorrow?"). Different loss functions imply different learning algorithms and, at times, very different predictors. While widespread and appealing, a clear drawback of this approach is that the loss function may not be known at the time of learning, requiring the algorithm to use a best-guess loss function. Alternatively, the same classifier may be used to inform multiple decisions, which correspond to multiple loss functions, requiring multiple learning algorithms to be run on the same data. We suggest a rigorous new paradigm for loss minimization in machine learning where the loss function can be ignored at the time of learning and only be taken into account when deciding an action.We introduce the notion of an (L, C)-omnipredictor, which could be used to optimize any loss in a family L. Once the loss function is set, the outputs of the predictor can be post-processed (a simple univariate data-independent transformation of individual predictions) to do well compared with any hypothesis from the class C. The post processing is essentially what one would perform if the outputs of the predictor were true probabilities of the uncertain events. In a sense, omnipredictors extract all the predictive power from the class C, irrespective of the loss function in L.We show that such "loss-oblivious" learning is feasible through a connection to multicalibration, a notion introduced in the context of algorithmic fairness. A multicalibrated predictor doesn't aim to minimize some loss function, but rather to make calibrated predictions, even when conditioned on inputs lying in certain sets c belonging to a family C which is weakly learnable. We show that a C-multicalibrated predictor is also an (L, C)-omnipredictor, where L contains all convex loss functions with some mild Lipschitz conditions. The predictors are even omnipredictors with respect to sparse linear combinations of functions in C. As a corollary, we deduce that distribution-specific weak agnostic learning is complete for a large class of loss minimization tasks.In addition, we show how multicalibration can be viewed as a solution concept for agnostic boosting, shedding new light on past results. Finally, we transfer our insights back to the context of algorithmic fairness by providing omnipredictors for multi-group loss minimization.

show abstract

Low-Degree Multicalibration

Gopalan¹,

Kim²,

Singhal³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Introduced as a notion of algorithmic fairness, multicalibration has proved to be a powerful and versatile concept with implications far beyond its original intent. This stringent notionthat predictions be well-calibrated across a rich class of intersecting subpopulations-provides its strong guarantees at a cost: the computational and sample complexity of learning multicalibrated predictors are high, and grow exponentially with the number of class labels. In contrast, the relaxed notion of multiaccuracy can be achieved more efficiently, yet many of the most desirable properties of multicalibration cannot be guaranteed assuming multiaccuracy alone. This tension raises a key question: Can we learn predictors with multicalibration-style guarantees at a cost commensurate with multiaccuracy?In this work, we define and initiate the study of Low-Degree Multicalibration. Low-Degree Multicalibration defines a hierarchy of increasingly-powerful multi-group fairness notions that spans multiaccuracy and the original formulation of multicalibration at the extremes. Our main technical contribution demonstrates that key properties of multicalibration, related to fairness and accuracy, actually manifest as low-degree properties. Importantly, we show that low-degree multicalibration can be significantly more efficient than full multicalibration. In the multi-class setting, the sample complexity to achieve low-degree multicalibration improves exponentially (in the number of classes) over full multicalibration. Our work presents compelling evidence that low-degree multicalibration represents a sweet spot, pairing computational and sample efficiency with strong fairness and accuracy guarantees.

show abstract

Multicalibrated Partitions for Importance Weights

Cited by 3 publications

References 14 publications

KL Divergence Estimation with Multi-group Attribution

KL Divergence Estimation with Multi-group Attribution

Omnipredictors

Low-Degree Multicalibration

Contact Info

Product

Resources

About