The Manager in the Machine: Organizational Priorities Influence AI Programmer’s Ability to Design Fair Models

Abstract

Artificial Intelligence (AI) models can improve everyday life by informing our decisions. Yet, experts have warned that these useful tools hold consequential biases that can perpetuate inequality in our decision-making. Computer scientists have demonstrated how the data used to train these models hold bias, while psychologists have shown that programmers’ own biases influence how (or if) they address this bias in the data. Yet both bodies of research do not fully account for an important fact: many of the programmers who construct these models are situated in organizations, and thus have to navigate organizational dynamics that influence how they construct AI models. We thus integrate theoretical concepts from computer science, psychology, and management to develop a novel theoretical model to illustrate how a construct developed in computer science – the “fairness vs. accuracy tradeoff” in model construction – impacts programmers’ decision-making when constructing AI models. We contend that programmers ultimately have to prioritize fairness or accuracy in their decision-making, and that they determine which priority to uphold based on their perceptions of their organizational leader’s priorities (a previously undertheorized dynamic in research on bias in machine learning). These perceptions shape their decision to express ethical voice about the model’s outcomes, an important predictor of the amount of bias the model holds. We further draw parallels between this tradeoff for organizations in developing AI and broader organizational contexts. This research contributes to management research by drawing further attention to an important organizational issue (developing fair AI models) using employee voice.