Measuring Fairness in Ranked Results

Raj, Amifa; Ekstrand, Michael D.

doi:10.1145/3477495.3532018

Cited by 34 publications

(10 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, as with users, we can also compute distributions of item-side effects such as exposure [28] over the various items or item providers (such as recording artists, film producers, or authors) and their attributes. This forms the basis of understanding how the benefits the system provides to the people who create and produce the items it recommends are distributed across those people both individually and with respect to socially-salient group identities [32,59].…”

Section: Itemsmentioning

confidence: 99%

Distributionally-Informed Recommender System Evaluation

Ekstrand,

Carterette,

Diaz

2024

ACM Trans. Recomm. Syst.

View full text Add to dashboard Cite

Current practice for evaluating recommender systems typically focuses on point estimates of user-oriented effectiveness metrics or business metrics, sometimes combined with additional metrics for considerations such as diversity and novelty. In this paper, we argue for the need for researchers and practitioners to attend more closely to various distributions that arise from a recommender system (or other information access system) and the sources of uncertainty that lead to these distributions. One immediate implication of our argument is that both researchers and practitioners must report and examine more thoroughly the distribution of utility between and within different stakeholder groups. However, distributions of various forms arise in many more aspects of the recommender systems experimental process, and distributional thinking has substantial ramifications for how we design, evaluate, and present recommender systems evaluation and research results. Leveraging and emphasizing distributions in the evaluation of recommender systems is a necessary step to ensure that the systems provide appropriate and equitably-distributed benefit to the people they affect.

show abstract

Section: Itemsmentioning

confidence: 99%

Distributionally-Informed Recommender System Evaluation

Ekstrand,

Carterette,

Diaz

2024

ACM Trans. Recomm. Syst.

View full text Add to dashboard Cite

show abstract

“…We consider two types of widely-used beyond-accuracy metrics, i.e., diversity and item fairness. Specifically, we investigate five fairness metrics (i.e., logEUR, logRUR, EEL, EED, and logDP) [28,36] and three diversity metrics (i.e., ILD, Entropy, and DS) [48]. To provide an overall understanding of these metrics, we group them according to different levels of connection with accuracy as follows: (i) Strong connection: logRUR, (ii) Weak connection: logEUR, EEL, EED (iii) No connection: logDP, ILD, Entropy, DS.…”

Section: Task Typementioning

confidence: 99%

“…2 Its expected value 𝜖 𝜋 = 𝐸 𝜋 𝜌 [𝜖 𝑃 ] is the group exposure among all the recommended baskets. Following [28,36], we select a set of well-known fairness metrics and cover two types of fairness considerations as follows: 3 (1) Equal opportunity. Promote equal treatment based on merit or utility, regardless of group membership [28,36].…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…Following [28,36], we select a set of well-known fairness metrics and cover two types of fairness considerations as follows: 3 (1) Equal opportunity. Promote equal treatment based on merit or utility, regardless of group membership [28,36]. (i) Exposed Utility Ratio (EUR) [38] quantifies the deviation from the objective that the exposure of each group is proportional to its utility 𝑌 (𝐺).…”

Section: Evaluation Metricsmentioning

confidence: 99%

See 1 more Smart Citation

A Next Basket Recommendation Reality Check

Jullien

Ariannezhad

et al. 2023

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

The goal of a next basket recommendation system is to recommend items for the next basket for a user, based on the sequence of their prior baskets. We examine whether the performance gains of the next basket recommendation (NBR) methods reported in the literature hold up under a fair and comprehensive comparison. To clarify the mixed picture that emerges from our comparison, we provide a novel angle on the evaluation of NBR methods, centered on the distinction between repetition and exploration: the next basket is typically composed of previously consumed items (i.e., repeat items) and new items (i.e., explore items). We propose a set of metrics that measure the repetition/exploration ratio and performance of NBR models. Using these new metrics, we provide a second analysis of state-of-the-art NBR models. The results help to clarify the extent of the actual progress achieved by existing NBR methods as well as the underlying reasons for any improvements that we observe. Overall, our work sheds light on the evaluation problem of NBR, provides a new evaluation protocol, and yields useful insights for the design of models for this task.

show abstract

“…In our experiments, we make use of the framework in [1], which uses two different notions for computing group fairness. However, other notions of group fairness can also be used, as long as they can be computed using the scores or ranks of workers with respect to jobs [2,28,31].…”

Section: Settingmentioning

confidence: 99%

A Framework to Maximize Group Fairness for Workers on Online Labor Platforms

et al. 2023

View full text Add to dashboard Cite

As the number of online labor platforms and the diversity of jobs on these platforms increase, ensuring group fairness for workers needs to be the focus of job-matching services. Risk of discrimination against workers occurs in two different job-matching services: when someone is looking for a job (i.e., a job seeker) and when someone wants to deploy jobs (i.e., a job provider). To maximize their chances of getting hired, job seekers submit their profiles on different platforms. Similarly, job providers publish their job offers on multiple platforms with the goal of reaching a wide and diverse workforce. In this paper, we propose a theoretical framework to maximize group fairness for workers 1) when job seekers are looking for jobs on multiple platforms, and 2) when jobs are being deployed by job providers on multiple platforms. We formulate each goal as different optimization problems with different constraints, prove most of them are computationally hard to solve and propose various efficient algorithms to solve all of them in reasonable time. We then design a series of experiments that rely on synthetic and semi-synthetic data generated from a real-world online labor platform to evaluate our framework.

show abstract

Measuring Fairness in Ranked Results

Cited by 34 publications

References 21 publications

Distributionally-Informed Recommender System Evaluation

Distributionally-Informed Recommender System Evaluation

A Next Basket Recommendation Reality Check

A Framework to Maximize Group Fairness for Workers on Online Labor Platforms

Contact Info

Product

Resources

About