How to Invest my Time

Zhang, Shanshan; He, Lihong; Dragut, Eduard C.; Vučetić, Slobodan

doi:10.1145/3292500.3330773

Cited by 26 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These determinations and associated terms and phrases served to develop the machine learning algorithm, which utilized a combination of humanin-the-loop and standard natural language processing and data analytics techniques [16,17].…”

Section: Curation and Symptom Term Table Generationmentioning

confidence: 99%

“…Natural language processing (NLP) classification [16,17] The data were first cleaned to identify spelling errors using a Java language-based package [18,19]. The advantage of using this package over Peter Norvig's algorithm-based Python autocorrect module [19] was that some of the specific jargon such as cramping and migraine that were incorrectly autocorrected to "tramping" and "migrate" by the Python algorithm were correctly retained by the Java module.…”

Section: Curation and Symptom Term Table Generationmentioning

confidence: 99%

See 1 more Smart Citation

What Patients Say: Large-Scale Analyses of Replies to the Parkinson’s Disease Patient Report of Problems (PD-PROP)

Marras

Arbatti²,

Hosamath³

et al. 2023

Journal of Parkinson’s Disease

View full text Add to dashboard Cite

Background: Free-text, verbatim replies in the words of people with Parkinson’s disease (PD) have the potential to provide unvarnished information about their feelings and experiences. Challenges of processing such data on a large scale are a barrier to analyzing verbatim data collection in large cohorts. Objective: To develop a method for curating responses from the Parkinson’s Disease Patient Report of Problems (PD-PROP), open-ended questions that asks people with PD to report their most bothersome problems and associated functional consequences. Methods: Human curation, natural language processing, and machine learning were used to develop an algorithm to convert verbatim responses to classified symptoms. Nine curators including clinicians, people with PD, and a non-clinician PD expert classified a sample of responses as reporting each symptom or not. Responses to the PD-PROP were collected within the Fox Insight cohort study. Results: Approximately 3,500 PD-PROP responses were curated by a human team. Subsequently, approximately 1,500 responses were used in the validation phase; median age of respondents was 67 years, 55% were men and median years since PD diagnosis was 3 years. 168,260 verbatim responses were classified by machine. Accuracy of machine classification was 95% on a held-out test set. 65 symptoms were grouped into 14 domains. The most frequently reported symptoms at first report were tremor (by 46% of respondents), gait and balance problems (>39%), and pain/discomfort (33%). Conclusion: A human-in-the-loop method of curation provides both accuracy and efficiency, permitting a clinically useful analysis of large datasets of verbatim reports about the problems that bother PD patients.

show abstract

Section: Curation and Symptom Term Table Generationmentioning

confidence: 99%

Section: Curation and Symptom Term Table Generationmentioning

confidence: 99%

What Patients Say: Large-Scale Analyses of Replies to the Parkinson’s Disease Patient Report of Problems (PD-PROP)

Marras

Arbatti²,

Hosamath³

et al. 2023

Journal of Parkinson’s Disease

View full text Add to dashboard Cite

show abstract

“…While many papers attempt to minimize labeling effort, a vast majority of them are measuring the effort by counting the number of labeled examples. There are very few papers (Zhang et al, 2019) that measure labeling effort in terms of elapsed time.…”

Section: Related Workmentioning

confidence: 99%

A Visualization Approach for Rapid Labeling of Clinical Notes for Smoking Status Extraction

Enayati¹,

Yang²,

Lu³

et al. 2021

Proceedings of the Second Workshop on Data Science With Human in the Loop: Language Advances

Self Cite

View full text Add to dashboard Cite

Labeling is typically the most human-intensive step during the development of supervised learning models. In this paper, we propose a simple and easy-to-implement visualization approach that reduces cognitive load and increases the speed of text labeling. The approach is fine-tuned for task of extraction of patient smoking status from clinical notes. The proposed approach consists of the ordering of sentences that mention smoking, centering them at smoking tokens, and annotating to enhance informative parts of the text. Our experiments on clinical notes from the MIMIC-III clinical database demonstrate that our visualization approach enables human annotators to label sentences up to 3 times faster than with a baseline approach.

show abstract

“…However, these neural models usually require exhaustive human efforts for generating labels for each token, and may not perform well in lowresource settings. To improve the performance of low-resource sequence labeling, several approaches have been applied including using semi-supervised methods Chen et al, 2020b), external weak supervision (Lison et al, 2020;Liang et al, 2020;Ren et al, 2020;Zhang et al, 2019;Yu et al, 2020) and active learning (Shen et al, 2017;Hazra et al, 2019;Liu et al, 2018;Fang et al, 2017;Gao et al, 2019). In this study, we mainly focus on active learning approaches which select samples based on the query policy design.…”

Section: Related Workmentioning

confidence: 99%

SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup

Zhang¹,

Yu²,

Zhang³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Active learning is an important technique for low-resource sequence labeling tasks. However, current active sequence labeling methods use the queried samples alone in each iteration, which is an inefficient way of leveraging human annotations. We propose a simple but effective data augmentation method to improve label efficiency of active sequence labeling. Our method, SeqMix, simply augments the queried samples by generating extra labeled sequences in each iteration. The key difficulty is to generate plausible sequences along with token-level labels. In SeqMix, we address this challenge by performing mixup for both sequences and token-level labels of the queried samples. Furthermore, we design a discriminator during sequence mixup, which judges whether the generated sequences are plausible or not. Our experiments on Named Entity Recognition and Event Detection tasks show that SeqMix can improve the standard active sequence labeling method by 2.27%-3.75% in terms of F 1 scores. The code and data for SeqMix can be found at https://github. com/rz-zhang/SeqMix.

show abstract

How to Invest my Time

Cited by 26 publications

References 19 publications

What Patients Say: Large-Scale Analyses of Replies to the Parkinson’s Disease Patient Report of Problems (PD-PROP)

What Patients Say: Large-Scale Analyses of Replies to the Parkinson’s Disease Patient Report of Problems (PD-PROP)

A Visualization Approach for Rapid Labeling of Clinical Notes for Smoking Status Extraction

SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup

Contact Info

Product

Resources

About