Learning classification models from multiple experts

Abstract

Building classification models from clinical data using machine learning methods often relies on labeling of patient examples by human experts. Standard machine learning framework assumes the labels are assigned by a homogeneous process. However, in reality the labels may come from multiple experts and it may be difficult to obtain a set of class labels everybody agrees on; it is not uncommon that different experts have different subjective opinions on how a specific patient example should be classified. In this work we propose and study a new multi-expert learning framework that assumes the class labels are provided by multiple experts and that these experts may differ in their class label assessments. The framework explicitly models different sources of disagreements and lets us naturally combine labels from different human experts to obtain: (1) a consensus classification model representing the model the group of experts converge to, as well as, and (2) individual expert models. We test the proposed framework by building a model for the problem of detection of the Heparin Induced Thrombocytopenia (HIT) where examples are labeled by three experts. We show that our framework is superior to multiple baselines (including standard machine learning framework in which expert differences are ignored) and that our framework leads to both improved consensus and individual expert models.

Active Learning of Classification Models with Likert-Scale Feedback

“…The clinical data consists of 50 patient state features essential for detection of HIT. The datasets consist of 579, 571, and 573 labeled patient state instances for Expert 1, 2 and 3 (see [30]), respectively. The labels include Likert-scale labels on four levels indicating the agreement, weak agreement, weak disagreement, and disagreement of the expert with the HIT alert.…”

Multidimensional Learning from Crowds: Usefulness and Application of Expertise Detection

“…was a procedure that searches for possible correlations between the labels of different experts. The self‐consistency of the labels provided by an expert allows Valizadegan et al . to identify labelers that introduce random noise, the least damaging kind of noise.…”

“…The main contribution of Wiebe et al 15 was a procedure that searches for possible correlations between the labels of different experts. The self-consistency of the labels provided by an expert allows Valizadegan et al 14 to identify labelers that introduce random noise, the least damaging kind of noise. The opposite strategy consists of learning a classifier from the annotations of each expert and, finally, combining all the models.…”

Hierarchical Active Learning with Proportion Feedback on Regions

“…It aims to sequentially select examples to be labeled next by evaluating the possible impact of the examples on the solution. Active learning has been successfully applied in domains as diverse as computer vision, natural language processing and bio-medical data mining [8,14,15]. …”