Towards Return Parity in Markov Decision Processes

Abstract

Algorithmic decisions made by machine learning models in high-stakes domains may have lasting impacts over time. Unfortunately, naive applications of standard fairness criterion in static settings over temporal domains may lead to delayed and adverse effects. To understand the dynamics of performance disparity, we study a fairness problem in Markov decision processes (MDPs). Specifically, we propose return parity, a fairness notion that requires MDPs from different demographic groups that share the same state and action spaces to achieve approximately the same expected time-discounted rewards. We first provide a decomposition theorem for return disparity, which decomposes the return disparity of any two MDPs into the distance between group-wise reward functions, the discrepancy of group policies, and the discrepancy between state visitation distributions induced by the group policies. Motivated by our decomposition theorem, we propose algorithms to mitigate return disparity via learning a shared group policy with state visitation distributional alignment using integral probability metrics. We conduct experiments to corroborate our results, showing that the proposed algorithm can successfully close the disparity gap while maintaining the performance of policies on two real-world recommender system benchmark datasets.

Achievement and Fragility of Long-term Equitability

“…However, dynamic effects of the fair decisions on evolving populations is a far less studied area. Recently, a series of papers, among which [9,11,12,15,19,21,35,47,49], have started to investigate these dynamic effects, e.g., by modeling decisions and dynamics as Markov Decision Processes and by using reinforcement learning as well as dynamic programming to design fair policies and algorithms. One of the main messages of these works is that fair decisions in a static context, may not be fair in a dynamic scenario, where populations and disadvantage groups evolve in response to the decisions taken at previous time periods.…”

“…Remark 5.1. This is quite interesting for funding allocation, but also in vaccine allocations over multiple periods, when one wants to make sure that the vaccination uptake is equivalent among neighboring countries (to allow for safe travel), and the global average uptake is as high as possible 9 . Figure 5 seems to indicate that the proportional-to-population strategy may not be as effective to achieve equitability as more targeted policies based on actual uptakes, and the countries' capacity to turn vaccine vials into vaccinated people.…”

“…Figure 5 seems to indicate that the proportional-to-population strategy may not be as effective to achieve equitability as more targeted policies based on actual uptakes, and the countries' capacity to turn vaccine vials into vaccinated people. 9 In this case, equitability represents equal uptake among neighboring countries.…”