Medical Relation Extraction with Manifold Models

Wang, Chang; Fan, Jialu

doi:10.3115/v1/p14-1078

Cited by 35 publications

(31 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The goal of our experiments is to assess the quality of our disagreement-aware crowdsourced data in training a medical relation extraction model. We use a binary classifier [27] that takes as input a set of sentences and two terms from the sentence, and returns a score reflecting the confidence of the model that a specific relation is expressed in the sentence between the terms. This manifold learning classifier was one of the first to accept weighted scores for each training instance, although it still requires a discrete positive or negative label.…”

Section: Methodsmentioning

confidence: 99%

Crowdsourcing Ground Truth for Medical Relation Extraction

Dumitrache

Aroyo

Welty

2018

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

Cognitive computing systems require human labeled data for evaluation and often for training. The standard practice used in gathering this data minimizes disagreement between annotators, and we have found this results in data that fails to account for the ambiguity inherent in language. We have proposed the CrowdTruth method for collecting ground truth through crowdsourcing, which reconsiders the role of people in machine learning based on the observation that disagreement between annotators provides a useful signal for phenomena such as ambiguity in the text. We report on using this method to build an annotated data set for medical relation extraction for the cause and treat relations, and how this data performed in a supervised training experiment. We demonstrate that by modeling ambiguity, labeled data gathered from crowd workers can (1) reach the level of quality of domain experts for this task while reducing the cost, and (2) provide better training data at scale than distant supervision. We further propose and validate new weighted measures for precision, recall, and F-measure, which account for ambiguity in both human and machine performance on this task.

show abstract

Section: Methodsmentioning

confidence: 99%

Crowdsourcing Ground Truth for Medical Relation Extraction

Dumitrache

Aroyo

Welty

2018

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…We train a medical relation extraction classifier [26] using both crowd results and expert judgments in a cross-validation experiment, and compare the results of the evaluation for each dataset using accuracy and F1 score. In a paper that is currently in submission, we prove that, in training the model, crowdsourced data from the lay crowd, that has been weighted with disagreement scores, performs just as well as gold standard data from medical experts.…”

Section: Preliminary Resultsmentioning

confidence: 99%

“…-Wikipedia medical sentences: relations and entities can be automatically collected with distant supervision [26] and the UMLS vocabulary of medical terms [7], but the data contains noise and requires human input for correction; -Wikipedia open domain sentences: relations and entities can be automatically collected with distant supervision [6] using DBpedia entities, also produces noisy data; -Twitter statuses: contain a variety of subjective opinions on current events, that can be retrieved based on hashtags.…”

Section: Research Methodology and Approachmentioning

confidence: 99%

Crowdsourcing Disagreement for Collecting Semantic Annotation

Dumitrache¹

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. This paper proposes an approach to gathering semantic annotation, which rejects the notion that human interpretation can have a single ground truth, and is instead based on the observation that disagreement between annotators can signal ambiguity in the input text, as well as how the annotation task has been designed. The purpose of this research is to investigate whether disagreement-aware crowdsourcing is a scalable approach to gather semantic annotation across various tasks and domains. We propose a methodology for answering this question that involves, for each task and domain: defining the crowdsourcing setup, experimental data collection, and evaluating both the setup and the results. We present initial results for the task of medical relation extraction, and propose an evaluation plan for crowdsourcing semantic annotation for several tasks and domains.

show abstract

“…dRiskKB is a KB constructed from MedLine using a semisupervised iterative pattern learning approach [16]. Wang et al [20] reported a KB extracted by a manifold medical relation extraction model.…”

Section: A Medical Knowledge Base Constructionmentioning

confidence: 99%

“…Semantic type information has been an important feature for identifying the relationship between two concepts in several existing automatic KB extraction studies [15], [18], [20]. This triggered us to utilize the semantic type and semantic group information in our model.…”

Section: Constructing Weighted Semantic Networkmentioning

confidence: 99%

Automatic Relationship Verification in Online Medical Knowledge Base: a Large Scale Study in SemMedDB

Zhang¹,

He²,

Zou³

et al. 2018

2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)

View full text Add to dashboard Cite

Automatically generated public medical knowledge bases (KBs), such as SemMedDB, are commonly used in various medical informatic tasks because of their comprehensive coverage. However, due to the imperfectness of the automatic algorithms for generating those KBs, they often contain noisy statements about medical concepts and relationships. For example, the extraction precision of SemRep, the tool used for constructing SemMedDB, is reported be 74.5%. Previous work focused on improving the algorithms for more accurate extraction. In this paper, however, we propose a supervised learning method to automatically verify the medical relationships. Through a study conducted on SemMedDB, we develop a method for generating a large set of training data with a relative small human labor annotation cost. We further propose nine features to characterize each medical relationship instance. After testing on several classifiers, our proposed methods can achieve the best F1 score and Accuracy at 80%, which demonstrates the effectiveness of our approach. In summary, our study demonstrates that noisy relationships in large scale medical KBs can be identified and removed without much human involvement.

show abstract

Medical Relation Extraction with Manifold Models

Cited by 35 publications

References 18 publications

Crowdsourcing Ground Truth for Medical Relation Extraction

Crowdsourcing Ground Truth for Medical Relation Extraction

Crowdsourcing Disagreement for Collecting Semantic Annotation

Automatic Relationship Verification in Online Medical Knowledge Base: a Large Scale Study in SemMedDB

Contact Info

Product

Resources

About