Automatically explaining machine learning prediction results: a demonstration on type 2 diabetes risk prediction

Luo, Gang

doi:10.1186/s13755-016-0015-4

Cited by 87 publications

(84 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…39 In addition, most machine learning models are complex and difficult to interpret because they depend heavily on aspects related to feature distribution, data availability and data representation. 40 In the present study we built and validated a simple and interpretable algorithm with excellent accuracy. Despite the high PPV and NPV in the stable, adequate glycaemic control trajectory, the PPV in the deteriorated glycaemic control trajectory was only 45.8% in the validation cohort.…”

Section: Discussionmentioning

confidence: 93%

“…One of the reasons for this could be that data obtained from EHRs are considered a byproduct of healthcare delivery, rather than a resource to improve its performance . In addition, most machine learning models are complex and difficult to interpret because they depend heavily on aspects related to feature distribution, data availability and data representation . In the present study we built and validated a simple and interpretable algorithm with excellent accuracy.…”

Section: Discussionmentioning

confidence: 98%

See 1 more Smart Citation

A risk score including body mass index, glycated haemoglobin and triglycerides predicts future glycaemic control in people with type 2 diabetes

Hertroijs

Elissen

Brouwers

et al. 2017

Diabetes Obesity Metabolism

View full text Add to dashboard Cite

AimTo identify, predict and validate distinct glycaemic trajectories among patients with newly diagnosed type 2 diabetes treated in primary care, as a first step towards more effective patient‐centred care.MethodsWe conducted a retrospective study in two cohorts, using routinely collected individual patient data from primary care practices obtained from two large Dutch diabetes patient registries. Participants included adult patients newly diagnosed with type 2 diabetes between January 2006 and December 2014 (development cohort, n = 10 528; validation cohort, n = 3777). Latent growth mixture modelling identified distinct glycaemic 5‐year trajectories. Machine learning models were built to predict the trajectories using easily obtainable patient characteristics in daily clinical practice.ResultsThree different glycaemic trajectories were identified: (1) stable, adequate glycaemic control (76.5% of patients); (2) improved glycaemic control (21.3% of patients); and (3) deteriorated glycaemic control (2.2% of patients). Similar trajectories could be discerned in the validation cohort. Body mass index and glycated haemoglobin and triglyceride levels were the most important predictors of trajectory membership. The predictive model, trained on the development cohort, had a receiver‐operating characteristic area under the curve of 0.96 in the validation cohort, indicating excellent accuracy.ConclusionsThe developed model can effectively explain heterogeneity in future glycaemic response of patients with type 2 diabetes. It can therefore be used in clinical practice as a quick and easy tool to provide tailored diabetes care.

show abstract

Section: Discussionmentioning

confidence: 93%

Section: Discussionmentioning

confidence: 98%

A risk score including body mass index, glycated haemoglobin and triglycerides predicts future glycaemic control in people with type 2 diabetes

Hertroijs

Elissen

Brouwers

et al. 2017

Diabetes Obesity Metabolism

View full text Add to dashboard Cite

show abstract

“…The clinical and administrative dataset is deidentified and publicly available from the Practice Fusion Diabetes Classification Challenge [ 15 , 34 ], containing 3-year (2009-2012) records as well as the labels of 9948 adult patients from all US states in the following year. A total of 1904 of these patients had a diagnosis of type 2 diabetes in the following year.…”

Section: Methodsmentioning

confidence: 99%

“…Historically, machine learning was blamed for being a black box. A recent method can automatically explain any machine learning model’s classification results with no accuracy loss [ 14 , 15 ]. Yet, two hurdles remain in using machine learning in health care.…”

Section: Introductionmentioning

confidence: 99%

Automating Construction of Machine Learning Models With Clinical Big Data: Proposal Rationale and Methods

et al. 2017

Self Cite

View full text Add to dashboard Cite

BackgroundTo improve health outcomes and cut health care costs, we often need to conduct prediction/classification using large clinical datasets (aka, clinical big data), for example, to identify high-risk patients for preventive interventions. Machine learning has been proposed as a key technology for doing this. Machine learning has won most data science competitions and could support many clinical activities, yet only 15% of hospitals use it for even limited purposes. Despite familiarity with data, health care researchers often lack machine learning expertise to directly use clinical big data, creating a hurdle in realizing value from their data. Health care researchers can work with data scientists with deep machine learning knowledge, but it takes time and effort for both parties to communicate effectively. Facing a shortage in the United States of data scientists and hiring competition from companies with deep pockets, health care systems have difficulty recruiting data scientists. Building and generalizing a machine learning model often requires hundreds to thousands of manual iterations by data scientists to select the following: (1) hyper-parameter values and complex algorithms that greatly affect model accuracy and (2) operators and periods for temporally aggregating clinical attributes (eg, whether a patient’s weight kept rising in the past year). This process becomes infeasible with limited budgets.ObjectiveThis study’s goal is to enable health care researchers to directly use clinical big data, make machine learning feasible with limited budgets and data scientist resources, and realize value from data.MethodsThis study will allow us to achieve the following: (1) finish developing the new software, Automated Machine Learning (Auto-ML), to automate model selection for machine learning with clinical big data and validate Auto-ML on seven benchmark modeling problems of clinical importance; (2) apply Auto-ML and novel methodology to two new modeling problems crucial for care management allocation and pilot one model with care managers; and (3) perform simulations to estimate the impact of adopting Auto-ML on US patient outcomes.ResultsWe are currently writing Auto-ML’s design document. We intend to finish our study by around the year 2022.ConclusionsAuto-ML will generalize to various clinical prediction/classification problems. With minimal help from data scientists, health care researchers can use Auto-ML to quickly build high-quality models. This will boost wider use of machine learning in health care and improve patient outcomes.

show abstract

“…Only a small number of the methods that are listed in ►Table 1 have been applied to predicting clinical outcomes. For example, Luo applied their method to type-2 diabetes risk prediction 18 , Štrumbelj et al developed and applied their method to breast cancer recurrence predictions, 19 and Reggia and Perricone developed explanations for predictions of the type of stroke. 11 More widespread application of these methods to clinical predictions can provide evidence of applicability and utility of these methods to clinical users.…”

Section: Background and Significancementioning

confidence: 99%

Patient-Specific Explanations for Predictions of Clinical Outcomes

et al. 2019

View full text Add to dashboard Cite

Background Machine learning models that are used for predicting clinical outcomes can be made more useful by augmenting predictions with simple and reliable patient-specific explanations for each prediction. Objectives This article evaluates the quality of explanations of predictions using physician reviewers. The predictions are obtained from a machine learning model that is developed to predict dire outcomes (severe complications including death) in patients with community acquired pneumonia (CAP). Methods Using a dataset of patients diagnosed with CAP, we developed a predictive model to predict dire outcomes. On a set of 40 patients, who were predicted to be either at very high risk or at very low risk of developing a dire outcome, we applied an explanation method to generate patient-specific explanations. Three physician reviewers independently evaluated each explanatory feature in the context of the patient's data and were instructed to disagree with a feature if they did not agree with the magnitude of support, the direction of support (supportive versus contradictory), or both. Results The model used for generating predictions achieved a F1 score of 0.43 and area under the receiver operating characteristic curve (AUROC) of 0.84 (95% confidence interval [CI]: 0.81–0.87). Interreviewer agreement between two reviewers was strong (Cohen's kappa coefficient = 0.87) and fair to moderate between the third reviewer and others (Cohen's kappa coefficient = 0.49 and 0.33). Agreement rates between reviewers and generated explanations—defined as the proportion of explanatory features with which majority of reviewers agreed—were 0.78 for actual explanations and 0.52 for fabricated explanations, and the difference between the two agreement rates was statistically significant (Chi-square = 19.76, p-value < 0.01). Conclusion There was good agreement among physician reviewers on patient-specific explanations that were generated to augment predictions of clinical outcomes. Such explanations can be useful in interpreting predictions of clinical outcomes.

show abstract

Automatically explaining machine learning prediction results: a demonstration on type 2 diabetes risk prediction

Cited by 87 publications

References 25 publications

A risk score including body mass index, glycated haemoglobin and triglycerides predicts future glycaemic control in people with type 2 diabetes

A risk score including body mass index, glycated haemoglobin and triglycerides predicts future glycaemic control in people with type 2 diabetes

Automating Construction of Machine Learning Models With Clinical Big Data: Proposal Rationale and Methods

Patient-Specific Explanations for Predictions of Clinical Outcomes

Contact Info

Product

Resources

About