Improving genomics-based predictions for precision medicine through active elicitation of expert knowledge

Sundin, Iiris; Peltola, Tomi; Micallef, Luana; Afrabandpey, Homayun; Soare, Marta; Majumder, Muntasir Mamun; Daee, Pedram; He, Chen; Serim, Barış; Havulinna, Aki S.; Heckman, Caroline A.; Marttinen, Pekka; Kaski, Samuel

doi:10.1093/bioinformatics/bty257

Cited by 17 publications

(18 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This creates a dependency between the feedback and training data that needs to be accounted for in the model to avoid double use of data and overfitting. the user (e.g., active learning) [3,18,19]. Some methods use validation datasets in addition to the training set to evaluate the performance.…”

Section: Updated Knowledgementioning

confidence: 99%

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

Daee

Peltola

Vehtari

et al. 2018

23rd International Conference on Intelligent User Interfaces

Self Cite

View full text Add to dashboard Cite

In human-in-the-loop machine learning, the user provides information beyond that in the training data. Many algorithms and user interfaces have been designed to optimize and facilitate this human-machine interaction; however, fewer studies have addressed the potential defects the designs can cause. Effective interaction often requires exposing the user to the training data or its statistics. The design of the system is then critical, as this can lead to double use of data and overfitting, if the user reinforces noisy patterns in the data. We propose a user modelling methodology, by assuming simple rational behaviour, to correct the problem. We show, in a user study with 48 participants, that the method improves predictive performance in a sparse linear regression sentiment analysis task, where graded user knowledge on feature relevance is elicited. We believe that the key idea of inferring user knowledge with probabilistic user models has general applicability in guarding against overfitting and improving interactive machine learning. * This is the pre-print version. The paper is published in the proceedings of IUI 2018 conference. Definitive

show abstract

Section: Updated Knowledgementioning

confidence: 99%

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

Daee

Peltola

Vehtari

et al. 2018

23rd International Conference on Intelligent User Interfaces

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [7], the authors proposed a method of knowledge elicitation for high-dimensional datasets, where an expert knows about the relevance of the covariates or values of the regression coefficients. A notable example of the practical knowledge elicitation applications for genomics prediction was proposed in [22].…”

Section: Related Workmentioning

confidence: 99%

Decision Rule Elicitation for Domain Adaptation

Nikitin

Kaski

2021

26th International Conference on Intelligent User Interfaces

Self Cite

View full text Add to dashboard Cite

Decision Rule Feedback Heuristical decision rule Feature space representationContinuous Improvement is not broken, because y > -1 y > -1 is broken?Could you check it?Figure 1: We consider a task where a machine learning model has been trained to predict breaks in a set of workstations. When the system receives new data about the workstations, it predicts their fault-risk and passes some of the predictions to the expert user for evaluation. Instead of just telling whether they agree, as in current human-in-the-loop systems, the expert gives a heuristic rule describing their decision-making. The machine learning system uses this rule to improve its decision making. Even though the rules will be imperfect and noisy, they will incorporate additional information available to the human users but not the system.

show abstract

“…In many applications, such as medical treatment effectiveness prediction (Sundin et al 2018), knowing the uncertainty in the prediction is important. Any explanation of the Fig.…”

Section: Interpreting Uncertaintymentioning

confidence: 99%

A decision-theoretic approach for model interpretability in Bayesian framework

Afrabandpey

Peltola

Piironen³

et al. 2020

Mach Learn

Self Cite

View full text Add to dashboard Cite

A salient approach to interpretable machine learning is to restrict modeling to simple models. In the Bayesian framework, this can be pursued by restricting the model structure and prior to favor interpretable models. Fundamentally, however, interpretability is about users’ preferences, not the data generation mechanism; it is more natural to formulate interpretability as a utility function. In this work, we propose an interpretability utility, which explicates the trade-off between explanation fidelity and interpretability in the Bayesian framework. The method consists of two steps. First, a reference model, possibly a black-box Bayesian predictive model which does not compromise accuracy, is fitted to the training data. Second, a proxy model from an interpretable model family that best mimics the predictive behaviour of the reference model is found by optimizing the interpretability utility function. The approach is model agnostic—neither the interpretable model nor the reference model are restricted to a certain class of models—and the optimization problem can be solved using standard tools. Through experiments on real-word data sets, using decision trees as interpretable models and Bayesian additive regression models as reference models, we show that for the same level of interpretability, our approach generates more accurate models than the alternative of restricting the prior. We also propose a systematic way to measure stability of interpretabile models constructed by different interpretability approaches and show that our proposed approach generates more stable models.

show abstract

Improving genomics-based predictions for precision medicine through active elicitation of expert knowledge

Cited by 17 publications

References 46 publications

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

Decision Rule Elicitation for Domain Adaptation

A decision-theoretic approach for model interpretability in Bayesian framework

Contact Info

Product

Resources

About