Evaluating Stochastic Rankings with Expected Exposure

Abstract

We introduce the concept of expected exposure as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item should receive more or less expected exposure than any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking. Leveraging user models from existing retrieval metrics, we propose a general evaluation methodology based on expected exposure and draw connections to related metrics in information retrieval evaluation. Importantly, this methodology relaxes classic information retrieval assumptions, allowing a system, in response to a query, to produce a distribution over rankings instead of a single fixed ranking. We study the behavior of the expected exposure metric and stochastic rankers across a variety of information access conditions, including ad hoc retrieval and recommendation. We believe that measuring and optimizing expected exposure metrics using randomization opens a new area for retrieval algorithm development and progress. CCS CONCEPTS • Information systems → Evaluation of retrieval results; Learning to rank.

Estimation of Fair Ranking Metrics with Incomplete Judgments

“…Morik et al [28] introduces a dynamic ranking scheme that optimizes the exposure metric introduced in [5]. Diaz et al [15] extend [5] to the context of stochastic ranking, including both individual and group fairness definitions. Pairwise rank fairness [2,24] does not directly measure exposure, but is related in that it measures the system's propensity to correctly or incorrectly rank relevant documents above irrelevant ones based on the relevant document's protected attribute: a system that is systematically more likely to correctly surface documents from the majority group than the protected group is deemed unfair.…”

“…Exposure-based representation measures the allocation of attention of searchers to items belonging to different groups [5,15,34]. Exposure is generally assumed to exponentially decrease with rank, albeit the exact formulations have differed in prior work [15]. There are some scenarios in IR tasks where the fraction of a particular group is the same for all systems.…”

“…Information retrieval (IR) evaluation often focuses on the effectiveness of a system, but there is also a significant history of measuring additional properties of a system's output or behavior, such as novelty and diversity [13]. In the last several years, there has been increased interest in the fairness of information retrieval systems [31], with a number of metrics being proposed [2,5,15,34,41,44]. While fairness metrics and constructs come in different flavors and aim at different goals, they all attempt to measure the societal impacts of the system decisions, and in particular to ensure that those impacts are equitably distributed.…”

“…This means that the fairness goal is to ensure that the providers of items or documents are treated fairly (as opposed to consumers or other stakeholders in multi-sided information access [8]). One way to evaluate this is by measuring whether the exposure different providers receive from the system is equitably distributed among providers of documents with comparable relevance [5,15]. In this spirit, one class of metrics seeks to ensure that socially-salient groups of providers receive comparable exposure; that is, to measure whether providers of, for example, a particular gender, ethnicity, professional seniority, or other group potentially subject to discrimination are systematically disadvantaged in the system's results.…”

“…To our knowledge, this is the first application of information retrieval metric estimation from incomplete data to the problem of assessing system fairness. In particular, we show how unbiased estimates of a few proportion-based metrics [41] and an exposure based metric [15] can be approximated with a significantly smaller number of group membership annotations. While we focus on these particular metrics in this paper, the techniques can easily be extended to estimate other fairness metrics when group membership annotations are incomplete.…”