Fair Top-k Ranking with multiple protected groups

Zehlike, Meike; Sühr, Tom; Baeza–Yates, Ricardo; Bonchi, Francesco; Castillo, Carlos; Hajian, Sara

doi:10.1016/j.ipm.2021.102707

Cited by 46 publications

(46 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One branch of the literature [172,174,63,30,8] reasons about probability-based fairness in the top-k ranking positions, which puts the focus onto group fairness. These works commonly provide a minimum (and for some cases also maximum) number or proportion of items/individuals from a protected groups, that are to be distributed evenly across the ranking.…”

Section: Overview Of Fair Ranking Literaturementioning

confidence: 99%

“…Thus, as part of the burgeoning algorithmic fairness literature [109,113], there have recently been many works on fairness in ranking, recommendation, and constrained allocation more broadly [25,172,174,63,30,8,147,19,152,70,26]. For example, suppose that the platform is deciding how to rank 10 items on a product search result page, and each item has demographic characteristics (such as those of the seller).…”

Section: Introductionmentioning

confidence: 99%

“…Then-in addition to considering each item's relevance-how should the platform rank the items, in a manner that is "fair" to the providers, either on an individual or group level? This question is often considered on an abstract level, independent of the specific ranking context; moreover, the literature primarily focuses on fairness of one instance of the ranking [172,171,174,147], or multiple independent instances of rankings with an additive objective across instances [19,150].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Fair ranking: a critical review, challenges, and future directions

Patro¹,

Porcaro²,

Mitchell³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Ranking, recommendation, and retrieval systems are widely used in online platforms and other societal systems, including e-commerce, media-streaming, admissions, gig platforms, and hiring. In the recent past, a large "fair ranking" research literature has been developed around making these systems fair to the individuals, providers, or content that are being ranked. Most of this literature defines fairness for a single instance of retrieval, or as a simple additive notion for multiple instances of retrievals over time. This work provides a critical overview of this literature, detailing the often context-specific concerns that such an approach misses: the gap between high ranking placements and true provider utility, spillovers and compounding effects over time, induced strategic incentives, and the effect of statistical uncertainty. We then provide a path forward for a more holistic and impact-oriented fair ranking research agenda, including methodological lessons from other fields and the role of the broader stakeholder community in overcoming data bottlenecks and designing effective regulatory environments.

show abstract

Section: Overview Of Fair Ranking Literaturementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Fair ranking: a critical review, challenges, and future directions

Patro¹,

Porcaro²,

Mitchell³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, empirical work typically shows bar graphs or tables to illustrate model performance disparities over multiple demographic groups. Although this granularity of information is undoubtedly useful and important on its own, these works do not provide a way to quickly synthesize group disparities of model performance across many potential models [1,12,30,52].…”

Section: Related Workmentioning

confidence: 99%

De-biasing "bias" measurement

Lum,

Zhang,

Bower

2022

Preprint

View full text Add to dashboard Cite

When a model's performance differs across socially or culturally relevant groups-like race, gender, or the intersections of many such groups-it is often called "biased." While much of the work in algorithmic fairness over the last several years has focused on developing various definitions of model fairness (the absence of group-wise model performance disparities) and eliminating such "bias, " much less work has gone into rigorously measuring it. In practice, it important to have high quality, human digestible measures of model performance disparities and associated uncertainty quantification about them that can serve as inputs into multi-faceted decision-making processes. In this paper, we show both mathematically and through simulation that many of the metrics used to measure group-wise model performance disparities are themselves statistically biased estimators of the underlying quantities they purport to represent. We argue that this can cause misleading conclusions about the relative group-wise model performance disparities along different dimensions, especially in cases where some sensitive variables consist of categories with few members. We propose the "double-corrected" variance estimator, which provides unbiased estimates and uncertainty quantification of the variance of model performance across groups. It is conceptually simple and easily implementable without statistical software package or numerical optimization. We demonstrate the utility of this approach through simulation and show on a real dataset that while statistically biased estimators of model group-wise model performance disparities indicate statistically significant between-group model performance disparities, when accounting for statistical bias in the estimator, the estimated group-wise disparities in model performance are no longer statistically significant.

show abstract

“…Often there are more relevant items than can be shown to the user, hence it is important to consider fairness of exposure for this set-up as well. Although there have been few approaches to fair top-𝑘 ranking [42,44], most are concerned with demographic parity, rather than merit-based fairness of exposure. Our contributions.…”

Section: Introductionmentioning

confidence: 99%

Fairness of Exposure in Light of Incomplete Exposure Estimation

Heuss,

Sarvi,

de Rijke

2022

Preprint

View full text Add to dashboard Cite

Fairness of exposure is a commonly used notion of fairness for ranking systems. It is based on the idea that all items or item groups should get exposure proportional to the merit of the item or the collective merit of the items in the group. Often, stochastic ranking policies are used to ensure fairness of exposure. Previous work unrealistically assumes that we can reliably estimate the expected exposure for all items in each ranking produced by the stochastic policy. In this work, we discuss how to approach fairness of exposure in cases where the policy contains rankings of which, due to inter-item dependencies, we cannot reliably estimate the exposure distribution. In such cases, we cannot determine whether the policy can be considered fair. Our contributions in this paper are twofold. First, we define a method called FELIX for finding stochastic policies that avoid showing rankings with unknown exposure distribution to the user without having to compromise user utility or item fairness. Second, we extend the study of fairness of exposure to the top-𝑘 setting and also assess FELIX in this setting. We find that FELIX can significantly reduce the number of rankings with unknown exposure distribution without a drop in user utility or fairness compared to existing fair ranking methods, both for full-length and top-𝑘 rankings. This is an important first step in developing fair ranking methods for cases where we have incomplete knowledge about the user's behaviour. CCS CONCEPTS• Information systems → Evaluation of retrieval results; Retrieval models and ranking.

show abstract

Fair Top-k Ranking with multiple protected groups

Cited by 46 publications

References 22 publications

Fair ranking: a critical review, challenges, and future directions

Fair ranking: a critical review, challenges, and future directions

De-biasing "bias" measurement

Fairness of Exposure in Light of Incomplete Exposure Estimation

Contact Info

Product

Resources

About