Explainability in human–agent systems

Rosenfeld, Avi; Richardson, Ariella

doi:10.1007/s10458-019-09408-y

Cited by 172 publications

(127 citation statements)

References 84 publications

Supporting

Mentioning

123

Contrasting

Unclassified

Order By: Relevance

“…Because we aimed to consider a variety of machine learning methods, we preferred using information gain and correlation-based feature selection, which are filter methods and can be used without any connection to a specific machine learning prediction model, 24 thus facilitating improved medical understanding. 35 We also considered correlations between features, which often exist in medical datasets. We used the multivariable correlation-based feature selection algorithm to do this.…”

Section: Discussionmentioning

confidence: 99%

“…32 We also considered using deep neural networks, but given the lack of dimensionality of our data, these models are substantially less accurate and interpretable. [33][34][35] Although we considered several options for building a supervised prediction model, unless otherwise specified, we present the results from a logistic regression prediction model.…”

Section: Data Handling and Machine Learning Approachesmentioning

confidence: 99%

See 1 more Smart Citation

Development and validation of a risk prediction model to diagnose Barrett's oesophagus (MARK-BE): a case-control machine learning approach

Rosenfeld

Graham

Jevons

et al. 2020

The Lancet Digital Health

Self Cite

View full text Add to dashboard Cite

Background Screening for Barrett's oesophagus relies on endoscopy, which is invasive and few who undergo the procedure are found to have the condition. We aimed to use machine learning techniques to develop and externally validate a simple risk prediction panel to screen individuals for Barrett's oesophagus. MethodsIn this prospective study, machine learning risk prediction in Barrett's oesophagus (MARK-BE), we used data from two case-control studies, BEST2 and BOOST, to compile training and validation datasets. From the BEST2 study, we analysed questionnaires from 1299 patients, of whom 880 (67·7%) had Barrett's oesophagus, including 40 with invasive oesophageal adenocarcinoma, and 419 (32·3%) were controls. We randomly split (6:4) the cohort using a computer algorithm into a training dataset of 776 patients and a testing dataset of 523 patients. We compiled an external validation cohort from the BOOST study, which included 398 patients, comprising 198 patients with Barrett's oesophagus (23 with oesophageal adenocarcinoma) and 200 controls. We identified independently important diagnostic features of Barrett's oesophagus using the machine learning techniques information gain and correlationbased feature selection. We assessed multiple classification tools to create a multivariable risk prediction model. Internal validation of the model using the BEST2 testing dataset was followed by external validation using the BOOST external validation dataset. From these data we created a prediction panel to identify at-risk individuals.Findings The BEST2 study included 40 diagnostic features. Of these, 19 added information gain but after correlationbased feature selection only eight showed independent diagnostic value including age, sex, cigarette smoking, waist circumference, frequency of stomach pain, duration of heartburn and acidic taste, and taking antireflux medication, of which all were associated with increased risk of Barrett's oesophagus, except frequency of stomach pain, with was inversely associated in a case-control population. Logistic regression offered the highest prediction quality with an area under the receiver-operator curve (AUC) of 0·87 (95% CI 0·84-0·90; sensitivity set at 90%; specificity of 68%). In the testing dataset, AUC was 0·86 (0·83-0·89; sensitivity set at 90%; specificity of 65%). In the external validation dataset, the AUC was 0·81 (0·74-0·84; sensitivity set at 90%; specificity of 58%). Interpretation Our diagnostic model offers valid predictions of diagnosis of Barrett's oesophagus in patients withsymptomatic gastro-oesophageal reflux disease, assisting in identifying who should go forward to invasive confirmatory testing. Our predictive panel suggests that overweight men who have been taking antireflux medication for a long time might merit particular consideration for further testing. Our risk prediction panel is quick and simple to administer but will need further calibration and validation in a prospective study in primary care.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Data Handling and Machine Learning Approachesmentioning

confidence: 99%

Development and validation of a risk prediction model to diagnose Barrett's oesophagus (MARK-BE): a case-control machine learning approach

Rosenfeld

Graham

Jevons

et al. 2020

The Lancet Digital Health

Self Cite

View full text Add to dashboard Cite

show abstract

“…End-users need explanations for various reasons such as the verification of the output, learning from the system and improving its future operation [71]. Recent studies and surveys in this field explored the user experience facets of explanations such as the explanation goals, content and the different forms of presenting and communicating these explanations including natural language and charts [69,64,1]. However, an understanding of the existing research on the delivery methods and modalities is becoming also needed.…”

Section: Introductionmentioning

confidence: 99%

Explainable Recommendations in Intelligent Systems: Delivery Methods, Modalities and Risks

Naiseh

Jiang

et al. 2020

Research Challenges in Information Science

View full text Add to dashboard Cite

With the increase in data volume, velocity and types, intelligent human-agent systems have become popular and adopted in different application domains, including critical and sensitive areas such as health and security. Humans' trust, their consent and receptiveness to recommendations are the main requirement for the success of such services. Recently, the demand on explaining the recommendations to humans has increased both from humans interacting with these systems so that they make an informed decision and, also, owners and systems managers to increase transparency and consequently trust and users' retention. Existing systematic reviews in the area of explainable recommendations focused on the goal of providing explanations, their presentation and informational content. In this paper, we review the literature with a focus on two user experience facets of explanations; delivery methods and modalities. We then focus on the risks of explanation both on user experience and their decision making. Our review revealed that explanations delivery to end-users is mostly designed to be along with the recommendation in a push and pull styles while archiving explanations for later accountability and traceability is still limited. We also found that the emphasis was mainly on the benefits of recommendations while risks and potential concerns, such as over-reliance on machines, is still a new area to explore.

show abstract

“…Sharing goals drives the actor's behavior to be explainable (interpretable) with understandability [13], explicability [14], legibility [15], predictability [16], and transparency [17], while hiding goals drives the actor's behavior to be uninterpretable with obfuscation, deception, privacy, and security [18]. In the cooperative setting, the explainability and interpretability of behavior remain significantly challenging in developing human-aware AI agents [18] and human-agent systems [19]. Goal signaling assists behavior explanation in human-robot teaming [20].…”

Section: Introductionmentioning

confidence: 99%

Goal Recognition Control under Network Interdiction Using a Privacy Information Metric

Luo

Wei

et al. 2019

Symmetry

View full text Add to dashboard Cite

Goal recognition (GR) is a method of inferring the goals of other agents, which enables humans or AI agents to proactively make response plans. Goal recognition design (GRD) has been proposed to deliberately redesign the underlying environment to accelerate goal recognition. Along with the GR and GRD problems, in this paper, we start by introducing the goal recognition control (GRC) problem under network interdiction, which focuses on controlling the goal recognition process. When the observer attempts to facilitate the explainability of the actor’s behavior and accelerate goal recognition by reducing the uncertainty, the actor wants to minimize the privacy information leakage by manipulating the asymmetric information and delay the goal recognition process. Then, the GRC under network interdiction is formulated as one static Stackelberg game, where the observer obtains asymmetric information about the actor’s intended goal and proactively interdicts the edges of the network with a bounded resource. The privacy leakage of the actor’s actions about the real goals is quantified by a min-entropy information metric and this privacy information metric is associated with the goal uncertainty. Next in importance, we define the privacy information metric based GRC under network interdiction (InfoGRC) and the information metric based GRC under threshold network interdiction (InfoGRCT). After dual reformulating, the InfoGRC and InfoGRCT as bi-level mixed-integer programming problems, one Benders decomposition-based approach is adopted to optimize the observer’s optimal interdiction resource allocation and the actor’s cost-optimal path-planning. Finally, some experimental evaluations are conducted to demonstrate the effectiveness of the InfoGRC and InfoGRCT models in the task of controlling the goal recognition process.

show abstract

Explainability in human–agent systems

Cited by 172 publications

References 84 publications

Development and validation of a risk prediction model to diagnose Barrett's oesophagus (MARK-BE): a case-control machine learning approach

Development and validation of a risk prediction model to diagnose Barrett's oesophagus (MARK-BE): a case-control machine learning approach

Explainable Recommendations in Intelligent Systems: Delivery Methods, Modalities and Risks

Goal Recognition Control under Network Interdiction Using a Privacy Information Metric

Contact Info

Product

Resources

About