Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints

Abstract

Prediction models have been widely adopted as the basis for decisionmaking in domains as diverse as employment, education, lending, and health. Yet, few real world problems readily present themselves as precisely formulated prediction tasks. In particular, there are often many reasonable target variable options. Prior work has argued that this is an important and sometimes underappreciated choice, and has also shown that target choice can have a significant impact on the fairness of the resulting model. However, the existing literature does not offer a formal framework for characterizing the extent to which target choice matters in a particular task. Our work fills this gap by drawing connections between the problem of target choice and recent work on predictive multiplicity. Specifically, we introduce a conceptual and computational framework for assessing how the choice of target affects individuals' outcomes and selection rate disparities across groups. We call this multi-target multiplicity. Along the way, we refine the study of single-target multiplicity by introducing notions of multiplicity that respect resource constraints-a feature of many real-world tasks that isn't captured by existing notions of predictive multiplicity. We apply our methods on a healthcare dataset, and show that the level of multiplicity that stems from target variable choice can be greater than that stemming from nearly-optimal models of a single target.

Algorithms for College Admissions Decision Support: Impacts of Policy Change and Inherent Variability

“…Missing from these assessments is an examination of arbitrariness in college admissions. Prior work on model multiplicity has shown that predictive tasks can be satisfied by multiple models that are equally accurate (e.g., at identifying students with academic merit) but differ in terms of their individual predictions (e.g., whether a specific applicant is considered a top applicant) [16,69]. This means that any number of seemingly minor decisions made by a college admissions office -down to something as simple as how to sample training data -could impact an individual applicant's outcome even if it does not majorly affect the ability of the college to identify students with academic merit.…”