FairLens: Auditing black-box clinical decision support systems

Panigutti, Cecilia; Perotti, Alan; Panisson, André; Bajardi, Paolo; Pedreschi, Dino

doi:10.1016/j.ipm.2021.102657

Cited by 51 publications

(18 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore it is not possible to propose an evidential value to Sherlock audits. It could become a strong evidence, like DNA in paternity test 20 , or it could be a more contextual, secondary evidence as DNA in criminal proceedings 21 . 17 18 Article 11 of the proposal.…”

Section: Probative Value Of the Audit Resultsmentioning

confidence: 99%

Algorithmic audits of algorithms, and the law

Merrer¹,

Pons²,

Trédan³

2022

Preprint

View full text Add to dashboard Cite

Algorithmic decision making is now widespread, ranging from health care allocation to more common actions such as recommendation or information ranking. The aim to audit these algorithms has grown alongside. In this paper, we focus on external audits that are conducted by interacting with the user side of the target algorithm, hence considered as a black box. Yet, the legal framework in which these audits take place is mostly ambiguous to researchers developing them: on the one hand, the legal value of the audit outcome is uncertain; on the other hand the auditors' rights and obligations are unclear.The contribution of this paper is to articulate two canonical audit forms to law, to shed light on these aspects:• the first audit form (we coin the Bobby audit form) checks a predicate against the algorithm, while the second (Sherlock) is more loose and opens up to multiple investigations. We find that: Bobby audits are more amenable to prosecution, yet are delicate as operating on real user data. This can lead to reject by a court (notion of admissibility). Sherlock audits craft data for their operation, most notably to build surrogates of the audited algorithm. It is mostly used for acts for whistleblowing, as even if accepted as a proof, the evidential value will be low in practice.• these two forms require the prior respect of a proper right to audit, granted by law or by the platform being audited; otherwise the auditor will be also prone to prosecutions regardless of the audit outcome. This article thus highlights the relation of current audits with law, in order to structure the growing field of algorithm auditing.Society rules through law, so that law is supposed to hold service providers and their algorithms accountable. In particular, decision-making algorithms are now widespread [6]. They directly face users, and govern large portions of our lives (from apparently subtle decisions such as recommendations, to more life changing ones such criminal justice or health care allocation [14]). The legal perspective on algorithms, especially of online platforms, is evolving to strong legal frameworks. For example, at an european level, in order to protect the fundamental rights of european residents 1 , or to frame artificial intelligence systems 2 explicitly shows the willingness to better regulate algorithms.The IT perspective and the nascent field of audits. As computer scientists and engineers, we are used to design and develop algorithms that process information and that can have an important impact on society [11,19]. For fine tuning these, developers operate a controlled feedback loop on data fed as inputs to the algorithm, and the corresponding algorithm results (output accuracy for instance).Considering an exterior viewpoint (the viewpoint of users or regulators), that observes or audits the behavior of remote algorithms is less frequent. A so called black box approach on algorithms can be dated back to Moore's tests black box automata in 1956 [20]. Relatively recent and sporadic works instead placed this ...

show abstract

Section: Probative Value Of the Audit Resultsmentioning

confidence: 99%

Algorithmic audits of algorithms, and the law

Merrer¹,

Pons²,

Trédan³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This is a typical barrier in machine learning, where many models exhibit large subgroup performance gaps, for example, of up to 38% [50][51][52][53][54][55][56][57]. No existing tool for auditing model bias and fairness [58,59] has been applied to our models. Currently, it is unknown how our models perform on key patient subgroups defined by independent variables such as race, ethnicity, and insurance type.…”

Section: Gap 2: the Models Exhibit Large Performance Gaps When Applie...mentioning

confidence: 99%

“…Each subgroup is defined by one or more attribute values. Given a predictive model built on a training set, model performance on each subgroup on the test set is computed and shown [ 58 , 59 ]. Machine learning needs enough training data to work well.…”

Section: Machine Learning Technique For Automatically Raising Model P...mentioning

confidence: 99%

A Roadmap for Boosting Model Generalizability for Predicting Hospital Encounters for Asthma

Luo

2022

JMIR Med Inform

View full text Add to dashboard Cite

In the United States, ~9% of people have asthma. Each year, asthma incurs high health care cost and many hospital encounters covering 1.8 million emergency room visits and 439,000 hospitalizations. A small percentage of patients with asthma use most health care resources. To improve outcomes and cut resource use, many health care systems use predictive models to prospectively find high-risk patients and enroll them in care management for preventive care. For maximal benefit from costly care management with limited service capacity, only patients at the highest risk should be enrolled. However, prior models built by others miss >50% of true highest-risk patients and mislabel many low-risk patients as high risk, leading to suboptimal care and wasted resources. To address this issue, 3 site-specific models were recently built to predict hospital encounters for asthma, gaining up to >11% better performance. However, these models do not generalize well across sites and patient subgroups, creating 2 gaps before translating these models into clinical use. This paper points out these 2 gaps and outlines 2 corresponding solutions: (1) a new machine learning technique to create cross-site generalizable predictive models to accurately find high-risk patients and (2) a new machine learning technique to automatically raise model performance for poorly performing subgroups while maintaining model performance on other subgroups. This gives a roadmap for future research.

show abstract

“…DM based on association rules has been effectively utilized in a variety of industries, including commerce. If simply the existence or absence of an item is considered, the rule is referred to as a Boolean Association Rule (BAR) [14]. BAR processes data in a distinct and categorical manner.…”

Section: Association Rule Mining (Arm)mentioning

confidence: 99%

Data Mining Technology Used in an Internet of Things-Based Decision Support System for Information Processing Intelligent Manufacturing

Sulhi

2021

IJIIS Int. J. Informatics Inf. Syst.

View full text Add to dashboard Cite

In recent years, database technology has improved significantly, and database management systems have gained widespread adoption. As a result, the volume of data saved across numerous databases has increased exponentially. However, the vast majority of information is hidden beneath this mountain of data. The goal of this study is to get a comprehensive understanding of the decision information system employed in the Internet of Things for intelligent manufacturing data processing. The proposed Decision support system (DSS) information processing is accomplished through the use of an IoT-based intelligent manufacturing data mining model. Numerous DM algorithms that are frequently encountered are analyzed, including the ARS and Apriori Algorithm (AA). The Decision Tree data mining algorithm is investigated, as is the generation of several Decision Trees and the pruning algorithm for digital twins. The findings demonstrate that data mining technology is capable of analyzing statistical data from a variety of angles and perspectives by modeling, classifying, and grouping large amounts of data as well as discovering correlations between them. Additionally, statistical work involves the calculation of data and the use of their correlations to aid in decision analysis. The proposed theoretical framework demonstrates how DSS-integrated components can work cooperatively in Intelligent Manufacturing to define a stable data flow within the Internet of Things. Particular emphasis is placed on conceptualizing the decision support system's integrated performance.

show abstract

FairLens: Auditing black-box clinical decision support systems

Cited by 51 publications

References 61 publications

Algorithmic audits of algorithms, and the law

Algorithmic audits of algorithms, and the law

A Roadmap for Boosting Model Generalizability for Predicting Hospital Encounters for Asthma

Data Mining Technology Used in an Internet of Things-Based Decision Support System for Information Processing Intelligent Manufacturing

Contact Info

Product

Resources

About