Improving condition severity classification with an efficient active learning based framework

Nissim, Nir; Boland, Mary Regina; Tatonetti, Nicholas P.; Elovici, Yuval; Hripcsak, George; Shaḥar, Yuval; Moskovitch, Robert

doi:10.1016/j.jbi.2016.03.016

Cited by 22 publications

(14 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As every person suffers from his/her individual disease manifestation, regardless of the overall disease label, the process of identifying molecular pathway culprits is often complicated by (a) the idiosyncrasies of individual genomic and epigenomic background, (b) pathway involvement in multiple diseases, whose symptoms may overlap or not, and (c) differences in environmental effects. Note that breaking down a disease into individual symptoms could potentially facilitate mapping of pathways to disease [110,111]. The specifics of the protein interactions within and outside the pathway, and the effects of genome variation on the required quantities and activities of each of the pathway components, define the type and severity of the resulting disease.…”

Section: Precision Medicine: Proteins and Disease Mechanismsmentioning

confidence: 99%

Protein function in precision medicine: deep understanding with machine learning

2016

View full text Add to dashboard Cite

Precision medicine and personalized health efforts propose leveraging complex molecular, medical and family history, along with other types of personal data toward better life. We argue that this ambitious objective will require advanced and specialized machine learning solutions. Simply skimming some low-hanging results off the data wealth might have limited potential. Instead, we need to better understand all parts of the system to define medically relevant causes and effects: how do particular sequence variants affect particular proteins and pathways? How do these effects, in turn, cause the health or disease-related phenotype? Toward this end, deeper understanding will not simply diffuse from deeper machine learning, but from more explicit focus on understanding protein function, context-specific protein interaction networks, and impact of variation on both.

show abstract

Section: Precision Medicine: Proteins and Disease Mechanismsmentioning

confidence: 99%

Protein function in precision medicine: deep understanding with machine learning

2016

View full text Add to dashboard Cite

show abstract

“…To better understand our new Inter -labeler variability experiments with the CAESER-ALE framework, we first summarize the results of our original basic experiment , which are presented in greater detail in our previous papers (49, 76) and have been significantly expanded in the current work. We present a summary of the results for the accuracy, although in our previous paper additional evaluation measures were used including: TPR, FPR, and AUC, as well as the number of new severe conditions discovered and acquired into the training set during each trial (49, 76).…”

Section: Resultsmentioning

confidence: 99%

“…This paper is based on the CAESAR-ALE framework which we developed in our recent study (49, 76). In this study, we aimed at comparing our AL methods to existing AL methods.…”

Section: Methodsmentioning

confidence: 99%

“…We have recently developed and assessed an Active Learning Enhancement version of CAESAR , called CAESAR-ALE , which was initially published as a preliminary study (49), and was then extended into a more detailed paper (76). Using three different AL methods, including two new AL methods that we developed, we demonstrated that the labeling burden on medical experts can be significantly reduced.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Inter-labeler and intra-labeler variability of condition severity classification models using active and passive learning methods

Nissim

Shaḥar

Elovici

et al. 2017

Artificial Intelligence in Medicine

Self Cite

View full text Add to dashboard Cite

Background and Objectives Labeling instances by domain experts for classification is often time consuming and expensive. To reduce such labeling efforts, we had proposed the application of active learning (AL) methods, introduced our CAESAR-ALE framework for classifying the severity of clinical conditions, and shown its significant reduction of labeling efforts. The use of any of three AL methods (one well known [SVM-Margin], and two that we introduced [Exploitation and Combination_XA]) significantly reduced (by 48% to 64%) condition labeling efforts, compared to standard passive (random instance-selection) SVM learning. Furthermore, our new AL methods achieved maximal accuracy using 12% fewer labeled cases than the SVM-Margin AL method. However, because labelers have varying levels of expertise, a major issue associated with learning methods, and AL methods in particular, is how to best to use the labeling provided by a committee of labelers. First, we wanted to know, based on the labelers’ learning curves, whether using AL methods (versus standard passive learning methods) has an effect on the Intra-labeler variability (within the learning curve of each labeler) and inter-labeler variability (among the learning curves of different labelers). Then, we wanted to examine the effect of learning (either passively or actively) from the labels created by the majority consensus of a group of labelers. Methods We used our CAESAR-ALE framework for classifying the severity of clinical conditions, the three AL methods and the passive learning method, as mentioned above, to induce the classifications models. We used a dataset of 516 clinical conditions and their severity labeling, represented by features aggregated from the medical records of 1.9 million patients treated at Columbia University Medical Center. We analyzed the variance of the classification performance within (intra-labeler), and especially among (inter-labeler) the classification models that were induced by using the labels provided by seven labelers. We also compared the performance of the passive and active learning models when using the consensus label. Results The AL methods produced, for the models induced from each labeler, smoother Intra-labeler learning curves during the training phase, compared to the models produced when using the passive learning method. The mean standard deviation of the learning curves of the three AL methods over all labelers (mean: 0.0379; range: [0.0182 to 0.0496]), was significantly lower (p = 0.049) than the Intra-labeler standard deviation when using the passive learning method (mean: 0.0484; range: [0.0275 to 0.0724). Using the AL methods resulted in a lower mean Inter-labeler AUC standard deviation among the AUC values of the labelers’ different models during the training phase, compared to the variance of the induced models’ AUC values when using passive learning. The Inter-labeler AUC standard deviation, using the passive learning method (0.039), was almost twice as high as the Inter-labeler standard deviation us...

show abstract

“…Active learning has been applied to different tasks within the clinical domain, e.g., classifying clinical concepts based on their assertions [115,116], de-identifying clinical records [117], clinical text classification [104], and clinical concept extraction [118,174,176]. It has also been attempted on a variety of biomedical tasks (e.g., [180,181,182,183]), which is out of scope of this paper.…”

Section: Active Learning In Clinical Information Extractionmentioning

confidence: 99%

Active Learning for Concept Extraction from Clinical Free Text

Kholghi¹

View full text Add to dashboard Cite

An increasing volume of clinical free-text data, such as discharge summaries and progress reports, has been collected by hospitals and healthcare centres and stored electronically for further processing. Extracting structured clinical information from such unstructured text resources is necessary for enabling secondary usage of reports, such as reporting, reasoning and retrieving, and for further processing in down-stream eHealth workflows. However, this analysis cannot be done manually, due to the high cost incurred by qualified experts to annotate the clinical free text. A significant initial step in extracting information from clinical free text is concept extraction, which involves identifying entities of interest in the clinical domain (such as diseases, medications, and symptoms). Currently, supervised machine learning approaches effectively extract clinical concepts by building powerful statistical models. However, these approaches require a large amount of high quality, annotated train data, which is created manually by domain experts through a costly and timeconsuming process. This results in a robust active learning framework for extracting clinical concepts, using state-of-the-art, active learning approaches. The second step is to leverage clinical information resources (i.e., terminologies and clinical information extraction tools) and other machine learning approaches (i.e., semi-supervised learning, unsupervised learning, and representation learning) to develop domain-specific and generic active learning approaches. This leads to a number of novel, active learning query strategies and a seed selection approach that outperform the state-of-the-art approaches with less manual annotation effort. The last step is to validate the benefits of the developed AL-based framework in reducing the annotation cost (i.e., time) through a comprehensive user study. An AL-assisted pre-annotation scheme is also introduced, in which the learning models built across the AL process generate high quality pre-annotations to be reviewed by human annotators. This further accelerates the annotation process, by significantly reducing the number of manual annotations that must be added or corrected compared to de novo annotation.The results of this study demonstrate that AL plays an important role in reducing the manual annotation cost. The CEAL framework extracts high quality domain concepts from clinical narratives, while significantly reducing the labour cost with up to 35% less annotation time required. Additionally, AL-assisted preannotations accelerate the de novo annotation process with a further 20% less annotation time required. This thesis contributes to information extraction from clinical unstructured text resources by alleviating the burden of manual annotation.The practical significance of this research is three-fold: (1) benefitting the overall patient healthcare by facilitating downstream eHealth workflows such as supporting clinical information processing, reporting, reasoning, and efficient decision m...

show abstract

Improving condition severity classification with an efficient active learning based framework

Cited by 22 publications

References 54 publications

Protein function in precision medicine: deep understanding with machine learning

Protein function in precision medicine: deep understanding with machine learning

Inter-labeler and intra-labeler variability of condition severity classification models using active and passive learning methods

Active Learning for Concept Extraction from Clinical Free Text

Contact Info

Product

Resources

About