Linear Classifier with Reject Option for the Detection of Vocal Fold Paralysis and Vocal Fold Edema

Kotropoulos, Constantine; Arce, Gonzalo R.

doi:10.1155/2009/203790

Cited by 14 publications

(2 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The relative preference for abstaining compared to making an incorrect classification is reflected in the results of the cost survey. Other examples of selective prediction exist; for example, Kotropoulos and Arce reported building a linear classifier with the option to postpone decision-making, which they called a rejection 24,25 . In this paper, postponed decisions required further advice by a domain expert in the context of the paper’s intended application.…”

Section: Discussionmentioning

confidence: 99%

Selective prediction for extracting unstructured clinical data

Swaminathan

López

Wang

et al. 2022

Preprint

View full text Add to dashboard Cite

Background: Electronic health records represent a large data source for outcomes research, but the majority of EHR data is unstructured (e.g. free text of clinical notes) and not conducive to computational methods. While there are currently approaches to handle unstructured data, such as manual abstraction, structured proxy variables, and model-assisted abstraction, these methods are time-consuming, not scalable, and require clinical domain expertise. This paper aims to determine whether selective prediction, which gives a model the option to abstain from generating a prediction, can improve the accuracy and efficiency of unstructured clinical data abstraction. Methods: We trained selective prediction models to identify the presence of four distinct clinical variables in free-text pathology reports: primary cancer diagnosis of glioblastoma (GBM, n = 659), resection of rectal adenocarcinoma (RRA, n = 601), and two procedures for resection of rectal adenocarcinoma: abdominoperineal resection (APR, n = 601) and low anterior resection (LAR, n = 601). Data were manually abstracted from pathology reports and used to train L1-regularized logistic regression models using term-frequency-inverse-document-frequency features. Data points that the model was unable to predict with high certainty were manually abstracted. Findings: All four selective prediction models achieved a test-set sensitivity, specificity, positive predictive value, and negative predictive value above 0.91. The use of selective prediction led to sizable gains in automation (anywhere from 57% to 95% reduction in manual abstraction of charts across the four outcomes). For our GBM classifier, the selective prediction model saw improvements to sensitivity (0.94 to 0.96), specificity (0.79 to 0.96), PPV (0.89 to 0.98), and NPV (0.88 to 0.91) when compared to a non-selective classifier. Interpretation: Selective prediction using utility-based probability thresholds can facilitate unstructured data extraction by giving "easy" charts to a model and "hard" charts to human abstractors, thus increasing efficiency while maintaining or improving accuracy.

show abstract

Section: Discussionmentioning

confidence: 99%

Selective prediction for extracting unstructured clinical data

Swaminathan

López

Wang

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The concept of a reject option in classifiers has been used in various machine-learning problems including gene-expression classification [13], odor classification [14], and vocal fold paralysis [15]. Studies have included a reject option in binary [16], multi-class [17], and multistage [9] classifications.…”

Section: Related Workmentioning

confidence: 99%

Speech Emotion Recognition with a Reject Option

Sridhar

Busso

2019

Interspeech 2019

View full text Add to dashboard Cite

Speech emotion recognition (SER) for categorical descriptors is a difficult task when the recordings come from everyday spontaneous interactions. The boundaries between emotional classes are less clear, resulting in complex, mixed emotions. Since the performance of a SER system varies across speech recordings, it is important to understand the reliability associated with its prediction. An intriguing formulation in machine learning related to this problem is the reject option, where a classifier only provides predictions over samples with reliability above a given threshold. This paper proposes a classification technique with a reject option using deep neural networks (DNNs) that increases its performance by selectively trading its coverage in the testing set. We use two different criteria to develop a SER system with a reject option, where it can accept or reject a sample as needed. Using the MSP-Podcast corpus, we evaluate this idea by comparing different classification performance as a function of coverage. By selectively defining a coverage of 75% of the samples, we obtain relative gains in F1-score of up to 25.71% for a five-class problem and 20.63% for an eight-class problem. The sentences that are rejected are analyzed in the evaluation, confirming that they have lower inter-evaluator agreement.

show abstract