Validating Generic Metrics of Fairness in Game-Based Resource Allocation Scenarios with Crowdsourced Annotations

Abstract

Being able to effectively measure the notion of fairness is of vital importance as it can provide insight into the formation and evolution of complex patterns and phenomena, such as social preferences, collaboration, group structures and social conflicts. This paper presents a comparative study for quantitatively modelling the notion of fairness in one-to-many resource allocation scenarios-i.e. one provider agent has to allocate resources to multiple receiver agents. For this purpose, we investigate the efficacy of six metrics and cross-validate them on crowdsourced human ranks of fairness annotated through a computer game implementation of the one-to-many resource allocation scenario. Four of the fairness metrics examined are well-established metrics of data dispersion, namely standard deviation, normalised entropy, the Gini coefficient and the fairness index. The fifth metric, proposed by the authors, is an ad-hoc context-based measure which is based on key aspects of distribution strategies. The sixth metric, finally, is machine learned via ranking support vector machines (SVMs) on the crowdsourced human perceptions of fairness. Results suggest that all ad-hoc designed metrics correlate well with the human notion of fairness, and the context-based metrics we propose appear to have a predictability advantage over the other ad-hoc metrics. On the other hand, the normalised entropy and fairness index metrics appear to be the most expressive and generic for measuring fairness for the scenario adopted in this study and beyond. The SVM model can automatically model fairness more accurately than any ad-hoc metric examined (with an accuracy of 81.86%) but it is limited by its expressivity and generalisability.

Managing bias and unfairness in data for decision support: a survey of machine learning and data engineering approaches to identify and mitigate bias and unfairness within data management and analytics systems

“…While there exists all these mathematical fairness definitions and metrics, they tend to be conflicting and it is impossible to comply with all of them simultaneously, as shown by Chouldechova et al [38]. Consequently, few papers [18,62,105,106,195] study how the fairness of data-driven decisionsupport systems is perceived in order to choose the most relevant definitions taking into account stakeholders' preferences and mathematical trade-offs. Srivastava et al [173] show that one simple definition of fairness (demographic parity) solely matches the expectations of users of hypothetical systems.…”

“…Srivastava et al [173] show that one simple definition of fairness (demographic parity) solely matches the expectations of users of hypothetical systems. Conversely, Lee et al [105,106] and Grappiolo et al [62] show that different stakeholders might value different and possibly multiple notions of fairness (e.g. efficient, egalitarian, or equalitarian allocations).…”

Research on human resource allocation management of enterprise information project construction

“…Literature [12] related scholars focus on the research and analysis of human resource allocation models, and after introducing problems in the practice of static human resource models, they deeply analyze the factors influencing the dynamic allocation of human resources and construct the corresponding dynamic human resource allocation models after mastering their relationships with other resources. The literature [13] analyzes the internal logic of strategic HRM and the organizational performance of enterprises and establishes a relevant conceptual model by combining the examples of enterprise management.…”