Exploring the Limits of Simple Learners in Knowledge Distillation for Document Classification with DocBERT

Adhikari, Ashutosh; Ram, Achyudh; Tang, Rachel; Hamilton, William L.; Lin, Jimmy

doi:10.18653/v1/2020.repl4nlp-1.10

Cited by 33 publications

(34 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From a technical perspective, the current BERT pre-trained model has an input limit of 512 tokens. In order to process lengthy documents such as the “Warnings and Precautions” section containing hundreds to thousands of words, various solutions have been proposed, including i) text truncation and ii) text splitting combined with different pooling methods or Long Short-Term Memory networks ( Adhikari et al, 2019a , 2019b ; Sun et al, 2020 ). Such more complex model structures do not fit better the classification criteria for this study and complicate the model interpretation, as compared to a sentence classification-based model structure.…”

Section: Discussionmentioning

confidence: 99%

BERT-Based Natural Language Processing of Drug Labeling Documents: A Case Study for Classifying Drug-Induced Liver Injury Risk

Liu

et al. 2021

Front. Artif. Intell.

View full text Add to dashboard Cite

Background & Aims: The United States Food and Drug Administration (FDA) regulates a broad range of consumer products, which account for about 25% of the United States market. The FDA regulatory activities often involve producing and reading of a large number of documents, which is time consuming and labor intensive. To support regulatory science at FDA, we evaluated artificial intelligence (AI)-based natural language processing (NLP) of regulatory documents for text classification and compared deep learning-based models with a conventional keywords-based model.Methods: FDA drug labeling documents were used as a representative regulatory data source to classify drug-induced liver injury (DILI) risk by employing the state-of-the-art language model BERT. The resulting NLP-DILI classification model was statistically validated with both internal and external validation procedures and applied to the labeling data from the European Medicines Agency (EMA) for cross-agency application.Results: The NLP-DILI model developed using FDA labeling documents and evaluated by cross-validations in this study showed remarkable performance in DILI classification with a recall of 1 and a precision of 0.78. When cross-agency data were used to validate the model, the performance remained comparable, demonstrating that the model was portable across agencies. Results also suggested that the model was able to capture the semantic meanings of sentences in drug labeling.Conclusion: Deep learning-based NLP models performed well in DILI classification of drug labeling documents and learned the meanings of complex text in drug labeling. This proof-of-concept work demonstrated that using AI technologies to assist regulatory activities is a promising approach to modernize and advance regulatory science.

show abstract

Section: Discussionmentioning

confidence: 99%

BERT-Based Natural Language Processing of Drug Labeling Documents: A Case Study for Classifying Drug-Induced Liver Injury Risk

Liu

et al. 2021

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…Note that SnipBERT is very different from the standard BERT concatenation approach where individual sequences (snippets) are concatenated into a single document and fed into the model (Huang et al, 2019;Devlin et al, 2018;Beltagy et al, 2020;Adhikari et al). In contrast, SnipBERT processes each short snippet individually and aggregates them in an end-to-end manner.…”

Section: Methodsmentioning

confidence: 99%

An Interpretable End-to-end Fine-tuning Approach for Long Clinical Text

Huang,

Garapati,

Rich

2020

Preprint

View full text Add to dashboard Cite

Unstructured clinical text in EHRs contains crucial information for applications including decision support, trial matching, and retrospective research. Recent work has applied BERT-based models to clinical information extraction and text classification, given these models' state-of-the-art performance in other NLP domains. However, BERT is difficult to apply to clinical notes because it doesn't scale well to long sequences of text. In this work, we propose a novel fine-tuning approach called SnipBERT. Instead of using entire notes, SnipBERT identifies crucial snippets and then feeds them into a truncated BERT-based model in a hierarchical manner. Empirically, SnipBERT not only has significant predictive performance gain across three tasks but also provides improved interpretability, as the model can identify key pieces of text that led to its prediction.

show abstract

“…As our source of data, we choose the question-answer portions of U.S. congressional hearings (all in English) for several reasons: they contain political and societal controversy identifiable by crowdsourced workers, they have a strong signal of ambiguity as to the form and intent of the response, and the data is plentiful. 2 A dataset statement is in Appendix D.…”

Section: Datasetmentioning

confidence: 99%

Did they answer? Subjective acts and intents in conversational discourse

Ferracane¹,

Durrett²,

Li³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Discourse signals are often implicit, leaving it up to the interpreter to draw the required inferences. At the same time, discourse is embedded in a social context, meaning that interpreters apply their own assumptions and beliefs when resolving these inferences, leading to multiple, valid interpretations. However, current discourse data and frameworks ignore the social aspect, expecting only a single ground truth. We present the first discourse dataset with multiple and subjective interpretations of English conversation in the form of perceived conversation acts and intents. We carefully analyze our dataset and create computational models to (1) confirm our hypothesis that taking into account the bias of the interpreters leads to better predictions of the interpretations, (2) and show disagreements are nuanced and require a deeper understanding of the different contextual factors. We share our dataset and code at http://github.com/ elisaF/subjective_discourse.

show abstract

Exploring the Limits of Simple Learners in Knowledge Distillation for Document Classification with DocBERT

Cited by 33 publications

References 12 publications

BERT-Based Natural Language Processing of Drug Labeling Documents: A Case Study for Classifying Drug-Induced Liver Injury Risk

BERT-Based Natural Language Processing of Drug Labeling Documents: A Case Study for Classifying Drug-Induced Liver Injury Risk

An Interpretable End-to-end Fine-tuning Approach for Long Clinical Text

Did they answer? Subjective acts and intents in conversational discourse

Contact Info

Product

Resources

About