Improving Chinese Word Segmentation with Wordhood Memory Networks

Tian, Yuanhe; Yan, Shuicheng; Xia, Fei; Zhang, Tong; Yong-gang, Wang

doi:10.18653/v1/2020.acl-main.734

Cited by 82 publications

(58 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Extra knowledge (e.g., pre-trained embeddings (Song et al, 2017;Song and Shi, 2018; and pretrained models (Devlin et al, 2019;Diao et al, 2019)) can provide useful information and thus enhance model performance for many NLP tasks (Tian et al, 2020a,b,c). Specifically, memory and memory-augmented neural networks (Zeng et al, 2018;Santoro et al, 2018;Diao et al, 2020;Tian et al, 2020d) are another line of related research, which can be traced back to , which proposed memory networks to leverage extra information for question answering; then Sukhbaatar et al (2015) improved it with an end-to-end design to ensure the model being trained with less supervision. Particularly for Transformer, there are also memory-based methods proposed.…”

Section: Base+rm+mclnmentioning

confidence: 99%

Generating Radiology Reports via Memory-driven Transformer

Chen¹,

Yan²,

Chang³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

259

192

View full text Add to dashboard Cite

Medical imaging is frequently used in clinical practice and trials for diagnosis and treatment. Writing imaging reports is time-consuming and can be error-prone for inexperienced radiologists. Therefore, automatically generating radiology reports is highly desired to lighten the workload of radiologists and accordingly promote clinical automation, which is an essential task to apply artificial intelligence to the medical domain. In this paper, we propose to generate radiology reports with memorydriven Transformer, where a relational memory is designed to record key information of the generation process and a memory-driven conditional layer normalization is applied to incorporating the memory into the decoder of Transformer. Experimental results on two prevailing radiology report datasets, IU X-Ray and MIMIC-CXR, show that our proposed approach outperforms previous models with respect to both language generation metrics and clinical evaluations. Particularly, this is the first work reporting the generation results on MIMIC-CXR to the best of our knowledge. Further analyses also demonstrate that our approach is able to generate long reports with necessary medical terms as well as meaningful image-text attention mappings. 1

show abstract

Section: Base+rm+mclnmentioning

confidence: 99%

Generating Radiology Reports via Memory-driven Transformer

Chen¹,

Yan²,

Chang³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

259

192

View full text Add to dashboard Cite

show abstract

“…Tian et al [4] showed that the extra pre-trained BERT language model as encoder concatenating to the Conditional Random Fields(CRF) can achieve 98.28% to 98.40% on the MSR2005 Chinese word segment test dataset.…”

Section: Related Workmentioning

confidence: 99%

“…Cui et al [3] fine-tuned BERT and XLNet with extra 4.5 billion Chinese tokens, and improved the performance of many downstream tasks. Tian et al [4] showed that the extra pre-trained BERT language model as encoder concatenating to the Conditional Random Fields(CRF) can achieve 98.28% to 98.40% on the MSR2005 Chinese word segment test dataset.…”

Section: Related Workmentioning

confidence: 99%

Extra Large Sequence Transformer Model for Chinese Word Segment

Shen

2021

Preprint

View full text Add to dashboard Cite

Chinese word segment is widely studied in document analysis. The accuracy of the current popular word segment model, LSTM+CRF, is still not satisfactory. Models trained by the popular dataset often fails in the out-domain situation. In this paper, combining the Transformer-XL layer, the Fully-Connect layer, and the Conditional Random Field layer, the proposed model improved 3.23% in the macro-F1 score, comparing to the BERT+CRF model, on the MSR2005 Chinese word segment test dataset.

show abstract

“…However, for joint CWS and POS tagging, previous approaches to leveraging the n-gram features are limited to directly concatenating the n-gram embeddings with the input character embedding, where unimportant n-grams may mislead the model and result in incorrect predictions. Therefore, assigning appropriate weights to different n-grams regarding to their contexts is a potential effective solution (Higashiyama et al, 2019;Tian et al, 2020b) to the joint task and we propose to use multi-channel attention to tackle this mission. In detail, we first categorize n-grams by a specific metric, which in this study is either their frequencies or lengths and then model the grouped n-grams in separate channels of attentions.…”

Section: The Multi-channel Attentionsmentioning

confidence: 99%

Joint Chinese Word Segmentation and Part-of-speech Tagging via Multi-channel Attention of Character N-grams

Tian¹,

Yan²,

Xia³

2020

Proceedings of the 28th International Conference on Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Chinese word segmentation (CWS) and part-of-speech (POS) tagging are two fundamental tasks for Chinese language processing. Previous studies have demonstrated that jointly performing them can be an effective one-step solution to both tasks and this joint task can benefit from a good modeling of contextual features such as n-grams. However, their work on modeling such contextual features is limited to concatenating the features or their embeddings directly with the input embeddings without distinguishing whether the contextual features are important for the joint task in the specific context. Therefore, their models for the joint task could be misled by unimportant contextual information. In this paper, we propose a character-based neural model for the joint task enhanced by multi-channel attention of n-grams. In the attention module, n-gram features are categorized into different groups according to several criteria, and n-grams in each group are weighted and distinguished according to their importance for the joint task in the specific context. To categorize n-grams, we try two criteria in this study, i.e., n-gram frequency and length, so that n-grams having different capabilities of carrying contextual information are discriminatively learned by our proposed attention module. Experimental results on five benchmark datasets for CWS and POS tagging demonstrate that our approach outperforms strong baseline models and achieves state-of-the-art performance on all five datasets.

show abstract

Improving Chinese Word Segmentation with Wordhood Memory Networks

Cited by 82 publications

References 31 publications

Generating Radiology Reports via Memory-driven Transformer

Generating Radiology Reports via Memory-driven Transformer

Extra Large Sequence Transformer Model for Chinese Word Segment

Joint Chinese Word Segmentation and Part-of-speech Tagging via Multi-channel Attention of Character N-grams

Contact Info

Product

Resources

About