Justicia: A Stochastic SAT Approach to Formally Verify Fairness

Abstract

As a technology ML is oblivious to societal good or bad, and thus, the field of fair machine learning has stepped up to propose multiple mathematical definitions, algorithms, and systems to ensure different notions of fairness in ML applications. Given the multitude of propositions, it has become imperative to formally verify the fairness metrics satisfied by different algorithms on different datasets. In this paper, we propose a stochastic satisfiability (SSAT) framework, Justicia, that formally verifies different fairness measures of supervised learning algorithms with respect to the underlying data distribution. We instantiate Justicia on multiple classification and bias mitigation algorithms, and datasets to verify different fairness metrics, such as disparate impact, statistical parity, and equalized odds. Justicia is scalable, accurate, and operates on non-Boolean and compound sensitive attributes unlike existing distribution-based verifiers, such as FairSquare and VeriFair. Being distribution-based by design, Justicia is more robust than the verifiers, such as AIF360, that operate on specific test samples. We also theoretically bound the finite-sample error of the verified fairness measure.

Reflexiones sobre Justicia Social: Aportes de la Teología Católica y la Filosofía Personalista en Latinoamérica

“…Ahora bien, desde un enfoque histórico, el compromiso de la iglesia con la justicia social en el contexto latinoamericano, se manifestó con una significación histórica particular en la encíclica Rerum Novarum del papa León XIII, misma que fue escrita en 1891 en un contexto de tensiones sociales y grandes desigualdades inherentes al progreso industrial, y cuyas discusiones se mantienen vigentes y se alinean con pág. 9290 investigaciones contemporáneas que subrayan la necesidad de establecer sistemas justos y equitativos en diversos contextos sociales y económicos (Ghosh, Basu & Meel, 2020).…”

Beyond robustness: Resilience verification of tree-based classifiers

“…This property belongs to the class of individual fairness and we choose it for different reasons: it is intuitive, popular, and provides global fairness guarantees beyond the test data. Since individual fairness and group fairness deal with different concerns and rely on rather different technical tools, we consider verification of group fairness as complementary to our work [34], [35], [36], [37].…”

Unifying Decision and Function Queries in Stochastic Boolean Satisfiability

“…There are recent developments of specialized solvers Jiang 2017, 2018) designed for specific fragments of SSAT, and general solvers (Majercik and Boots 2005;Chen, Huang, and Jiang 2021;Wang et al 2022;Fan and Jiang 2023) that place no restrictions on the SSAT formula. Regarding applications, SSAT has been used in encoding problems, such as contingent planning (Majercik and Littman 2003), partially observable Markov decision processes (POMDPs) (Salmon and Poupart 2020), the fairness analysis of machine learning models (Ghosh, Basu, and Meel 2021), and probabilistic graphical models (Hsieh and Jiang 2022).…”