Semantic Textual Similarity in Japanese Clinical Domain Texts Using BERT

Abstract

Background Semantic textual similarity (STS) captures the degree of semantic similarity between texts. It plays an important role in many natural language processing applications such as text summarization, question answering, machine translation, information retrieval, dialog systems, plagiarism detection, and query ranking. STS has been widely studied in the general English domain. However, there exists few resources for STS tasks in the clinical domain and in languages other than English, such as Japanese. Objective The objective of this study is to capture semantic similarity between Japanese clinical texts (Japanese clinical STS) by creating a Japanese dataset that is publicly available. Materials We created two datasets for Japanese clinical STS: (1) Japanese case reports (CR dataset) and (2) Japanese electronic medical records (EMR dataset). The CR dataset was created from publicly available case reports extracted from the CiNii database. The EMR dataset was created from Japanese electronic medical records. Methods We used an approach based on bidirectional encoder representations from transformers (BERT) to capture the semantic similarity between the clinical domain texts. BERT is a popular approach for transfer learning and has been proven to be effective in achieving high accuracy for small datasets. We implemented two Japanese pretrained BERT models: a general Japanese BERT and a clinical Japanese BERT. The general Japanese BERT is pretrained on Japanese Wikipedia texts while the clinical Japanese BERT is pretrained on Japanese clinical texts. Results The BERT models performed well in capturing semantic similarity in our datasets. The general Japanese BERT outperformed the clinical Japanese BERT and achieved a high correlation with human score (0.904 in the CR dataset and 0.875 in the EMR dataset). It was unexpected that the general Japanese BERT outperformed the clinical Japanese BERT on clinical domain dataset. This could be due to the fact that the general Japanese BERT is pretrained on a wide range of texts compared with the clinical Japanese BERT.

An Entity Relationship Extraction Model Based on BERT-BLSTM-CRF for Food Safety Domain

“…Instead of the traditional one-way language model or the method of shallow splicing of two one-way language models for pretraining, the algorithm adopts the new marked language model, which can generate deep two-way language representation and fine-tune specific downstream tasks [ 22 ]. At present, BERT has played an essential role in the research field of natural language processing, such as entity relationship extraction, text emotion analysis, and text classification [ 23 ]. Gao et al [ 24 ] proposed a medical relationship extraction model based on BERT, which combined the whole sentence information obtained from the pretrained language model with the corresponding information of two medical entities to complete the relationship extraction task.…”

Large language models in textual analysis for gesture selection

“…BERT consists of transformer encoder layers and is designed for language modeling and next-sentence prediction. It has been shown to powerfully predict semantic similarities between word-and sentence-pairs [51,58] and this makes it a potentially valuable tool for gesture generation as mappings from words to gestures can be generalized over synonyms [23,43]. However, BERT does not have in-built explainable AI, limiting its application to realizing designer intent.…”

Uniting Multi-Scale Local Feature Awareness and the Self-Attention Mechanism for Named Entity Recognition

“…The Bert model uses bidirectional transformers as encoders, which enhances the generalization ability of the word vector model, fully describes the relationships between characters, words, and sentences, and effectively represents the semantic information between contexts. The Bert model has become the mainstream model in the field of NLP [31,32].…”