Biomedical-domain pre-trained language model for extractive summarization

Du, Yongping; Li, Qingxiao; Wang, Lulin; He, Yanqing

doi:10.1016/j.knosys.2020.105964

Cited by 49 publications

(13 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Du et al proposed a BioBERTSum [20]model to obtain sentence and token-level contextual representation. A domain-aware bidirectional language model pre-trained on a large-scale biomedical corpus is utilized to employ this model.…”

Section: B Other Approaches In Text Summarizationmentioning

confidence: 99%

OCR-based Hybrid Image Text Summarizer using Luhn Algorithm with FinetuneTransformer Modelsfor Long Document

Singco¹,

Trillo²,

Abalorio³

et al. 2023

IJETAE

View full text Add to dashboard Cite

The accessibility of an enormous number of image text documents on the internet has expanded the opportunities to develop a system for image text recognition with text summarization. Several approaches used in ATS in the literature are based on extractive and abstractive techniques; however, few implementations of the hybrid approach were observed. This paper employed state-of-the-art transformer models with the Luhn algorithm for extracted texts using Tesseract OCR. Nine models were generated and tested using the hybrid text summarization approach. Using ROUGE metrics, we compared the proposed system finetune abstractive models against existing abstractive models that use the same dataset Xsum. As a result, the finetune model got the highest ROUGE score during evaluation; in ROUGE-1 score was 57%, the ROUGE-2 score was 43%, and the ROUGE-L score was 42%. Furthermore, even when better algorithms and models were available for summarization, the Luhn algorithm and T5 finetune model provided significant results.

show abstract

Section: B Other Approaches In Text Summarizationmentioning

confidence: 99%

OCR-based Hybrid Image Text Summarizer using Luhn Algorithm with FinetuneTransformer Modelsfor Long Document

Singco¹,

Trillo²,

Abalorio³

et al. 2023

IJETAE

View full text Add to dashboard Cite

show abstract

“…Extractive summarization: Du et al [51], Moradi et al [162], Padmakumar et al [174], Kanwal et al [101], Dan et al [44], Song et al [220]. Abstractive summarization: Wallace et al [247], Gharebagh et al [62].…”

Section: Question Answeringmentioning

confidence: 99%

“…To explore the advanced pre-trained language models in the text summarizing of biomedical domain, the domain knowledge is incorporated by existing methods via domain fine-tuning. For biomedical extractive summarization, Du et al [51] proposed a novel model BioBERTSum, which used the domain-aware pre-trained language model as the encoder and then fine-tune it on the biomedical extractive summarization task. Gharebagh et al [62] utilized the domain knowledge-the salient medical ontological terms to help the content selection of the SciBERT based clinical abstractive summarization model.…”

Section: Text Summarizationmentioning

confidence: 99%

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Wang¹,

Xie²,

Pei³

et al. 2021

Preprint

View full text Add to dashboard Cite

Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities proposes various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.CCS Concepts: • Computing methodologies → Natural language processing; Natural language generation; Neural networks; Bio-inspired approaches.

show abstract

“…One is to convert the input text into character vectors; the other is to convert the input text into word vectors. In order to improve the performance of medical entity recognition, we use the pretraining model ELMO [43] to generate the required character and word vectors discussed in this paper.…”

Section: Input Layermentioning

confidence: 99%

Chinese Medical Entity Recognition Model Based on Character and Word Vector Fusion

Hou

Zhang

et al. 2021

Scientific Programming

View full text Add to dashboard Cite

The medical information carried in electronic medical records has high clinical research value, and medical named entity recognition is the key to extracting valuable information from large-scale medical texts. At present, most of the studies on Chinese medical named entity recognition are based on character vector model or word vector model. Owing to the complexity and specificity of Chinese text, the existing methods may fail to achieve good performance. In this study, we propose a Chinese medical named entity recognition method that fuses character and word vectors. The method expresses Chinese texts as character vectors and word vectors separately and fuses them in the model for features. The proposed model can effectively avoid the problems of missing character vector information and inaccurate word vector partitioning. On the CCKS 2019 dataset for the named entity recognition task of Chinese electronic medical records, the proposed model achieves good performance and can effectively improve the accuracy of Chinese medical named entity recognition compared with other baseline models.

show abstract

Biomedical-domain pre-trained language model for extractive summarization

Cited by 49 publications

References 19 publications

OCR-based Hybrid Image Text Summarizer using Luhn Algorithm with FinetuneTransformer Modelsfor Long Document

OCR-based Hybrid Image Text Summarizer using Luhn Algorithm with FinetuneTransformer Modelsfor Long Document

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Chinese Medical Entity Recognition Model Based on Character and Word Vector Fusion

Contact Info

Product

Resources

About