Crowdsourcing Perceptions of Fair Predictors for Machine Learning

Abstract

The increased reliance on algorithmic decision-making in socially impactful processes has intensified the calls for algorithms that are unbiased and procedurally fair. Identifying fair predictors is an essential step in the construction of equitable algorithms, but the lack of ground-truth in fair predictor selection makes this a challenging task. In our study, we recruit 90 crowdworkers to judge the inclusion of various predictors for recidivism. We divide participants across three conditions with varying group composition. Our results show that participants were able to make informed decisions on predictor selection. We find that agreement with the majority vote is higher when participants are part of a more diverse group. The presented workflow, which provides a scalable and practical approach to reach a diverse audience, allows researchers to capture participants' perceptions of fairness in private while simultaneously allowing for structured participant discussion.CCS Concepts: • Human-centered computing → Human computer interaction (HCI); Collaborative and social computing.

Designing Alternative Representations of Confusion Matrices to Support Non-Expert Public Understanding of Algorithm Performance

“…This body of research has used interviews, surveys, and experiments to empirically probe people's perceptions towards algorithmic decisions. Examples include studies on how humans perceive algorithmic decisions versus human decisions in managerial contexts [34], whether people perceive certain features (such as criminal history or neighborhood safety) as fair to be used to predict criminal risk [25,50], how explanation styles might matter in shaping people's justice perceptions [9], how members of traditionally marginalized communities feel about algorithm (un)fairness [54], how affected communities feel about algorithmic decisions in the context of a child welfare system [12], how the general public perceives online behavioral advertising that used demographic factors (e.g., race) as targeting variables [44], which statistical definitions of fairness people perceive to be the fairest in the context of loan decisions [47], as well as how humans use AI systems to make decisions [23,24]. This past body of work mostly used storyboards or text to present several algorithmic scenarios to their study participants, often without tackling the results and performance of the underlying machine learning models.…”

“…As a result, researchers in HCI are calling for more efforts to explore how to better explain and present algorithmic decisions to multiple stakeholders, especially to non-experts with limited technical literacy [58]. Recent studies (e.g., [13,35,50,56]) have started to use visualizations or create user interfaces to communicate algorithmic decisions to their study participants.…”

“…Past work in the HCI community has used text or storyboards to present different scenarios to their study participants in order to probe and understand how humans perceive algorithmic decisions (e.g., [9,25,34,54]). Emerging studies [35,50,56] started to use visualizations and create user interfaces to communicate algorithmic decisions to their study participants. Our study contributes to this quickly growing body of work by developing more intuitive and generally understandable representations of algorithmic performance.…”

Crowdsourcing sensitive data using public displays—opportunities, challenges, and considerations

“…Further studies (preferably outside the university environment) need to be conducted to ascertain whether our experiences obtained within the academic community apply outside the campus as well. The required human-centric viewpoint in sensitive data collection [5,51] has yet to be exhaustively discussed within the scope of public displays. We hope that our study is a call to action in itself concerning these aspects; How can we start leveraging public displays ethically for what they really excel in-collecting data serendipitously and largely on autopilot-for purposes beyond one single application at a time?…”

A Systematic Assessment of National Artificial Intelligence Policies: Perspectives from the Nordics and Beyond

“…In May 2019, forty-two countries adopted the first set of intergovernmental policy guidelines on AI as set forward by the OECD (Organisation for Economic Co-operation and Development) [34]. These guidelines promote concepts such as accountability, fairness, and transparency -as has been previously advocated within the field of HCI [1,44,48] and the wider Computer Science community [2].…”