How do fairness definitions fare? Testing public attitudes towards three algorithmic definitions of fairness in loan allocations

Saxena, Nripsuta; Huang, Karen; DeFilippis, Evan; Radanovic, Goran; Parkes, David C.; Liu, Yang

doi:10.1016/j.artint.2020.103238

Cited by 82 publications

(82 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Two additional empirical studies focus on questions complementary to ours. In a loan-allocation context, Saxena et al quantify attitudes about individual fairness [24]. They ask participants to rate the fairness of three different ways a loan officer could divide $50,000 between two individuals with different repayment rates (and in one iteration, different races).…”

Section: Empirical Studies Of Fairnessmentioning

confidence: 99%

An empirical study on the perceived fairness of realistic, imperfect machine learning models

Harrison

Hanson

Jacinto

et al. 2020

Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency

View full text Add to dashboard Cite

Section: Empirical Studies Of Fairnessmentioning

confidence: 99%

An empirical study on the perceived fairness of realistic, imperfect machine learning models

Harrison

Hanson

Jacinto

et al. 2020

Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency

View full text Add to dashboard Cite

“…There are several definitions of fairness that proposed in the literature but with no firm consensus being yet reached [191], [192], [193], [194]. Even though, these definitions can be used as oracles to detect fairness violations in ML testing.…”

Section: Fairness Definitions and Measurement Metricsmentioning

confidence: 99%

“…Saxena et al [193] investigated people's perceptions of three of the fairness definitions. About 200 recruited participants from Amazon's Mechanical Turk were asked to choose their preference over three allocation rules on two individuals having each applied for a loan.…”

Section: ) Group Fairnessmentioning

confidence: 99%

Machine Learning Testing: Survey, Landscapes and Horizons

Zhang

Harman

Ма

et al. 2022

IIEEE Trans. Software Eng.

Self Cite

534

318

View full text Add to dashboard Cite

This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 128 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing. Index Terms-machine learning, software testing, deep neural network, ! • Jie M. Zhang and Mark Harman are with CREST, University College London, United Kingdom. Mark Harman is also with Facebook London.

show abstract

“…To our knowledge, no prior work has conducted experiments with the goal of mapping existing group definitions of fairness to human perception of justice. Saxena et al [26] investigate ordinary people's attitude toward three notions of individual fairness in the context of loan decisions. They investigate the following three notions: 1) treating similar individuals similarly [9]; 2) never favoring a worse individual over a better one [18]; 3) the probability of approval being proportional to the chance of the individual representing the best choice.…”

Section: Related Workmentioning

confidence: 99%

Mathematical Notions vs. Human Perception of Fairness

Srivastava

Heidari

Krause

2019

Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

135

View full text Add to dashboard Cite

Fairness for Machine Learning has received considerable attention, recently. Various mathematical formulations of fairness have been proposed, and it has been shown that it is impossible to satisfy all of them simultaneously. The literature so far has dealt with these impossibility results by quantifying the tradeoffs between different formulations of fairness. Our work takes a different perspective on this issue. Rather than requiring all notions of fairness to (partially) hold at the same time, we ask which one of them is the most appropriate given the societal domain in which the decision-making model is to be deployed. We take a descriptive approach and set out to identify the notion of fairness that best captures lay people's perception of fairness. We run adaptive experiments designed to pinpoint the most compatible notion of fairness with each participant's choices through a small number of tests. Perhaps surprisingly, we find that the most simplistic mathematical definition of fairnessnamely, demographic parity-most closely matches people's idea of fairness in two distinct application scenarios. This conclusion remains intact even when we explicitly tell the participants about the alternative, more complicated definitions of fairness, and we reduce the cognitive burden of evaluating those notions for them. Our findings have important implications for the Fair ML literature and the discourse on formalizing algorithmic fairness.

show abstract

How do fairness definitions fare? Testing public attitudes towards three algorithmic definitions of fairness in loan allocations

Cited by 82 publications

References 15 publications

An empirical study on the perceived fairness of realistic, imperfect machine learning models

An empirical study on the perceived fairness of realistic, imperfect machine learning models

Machine Learning Testing: Survey, Landscapes and Horizons

Mathematical Notions vs. Human Perception of Fairness

Contact Info

Product

Resources

About