Glyph2Vec: Learning Chinese Out-of-Vocabulary Word Embedding from Glyphs

Chen, Hongyou; Yu, Sz-Han; Lin, Shou-De

doi:10.18653/v1/2020.acl-main.256

Cited by 12 publications

(8 citation statements)

References 18 publications

(27 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For completeness, we also replace the SVM classifier with a neural network classifier (Chen et al, 2020a) consisting of 3 fully-connected layers. The results are shown in Appendix I.…”

Section: Word Embedding Learningmentioning

confidence: 99%

Lifelong Learning of Topics and Domain-Specific Word Embeddings

Qin¹,

Lu²,

Rao³

2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Lifelong topic models mainly focus on indomain text streams in which each chunk only contains documents from a single domain. To overcome data diversity of the in-domain corpus, most of the existing methods exploit the information from limited sources in a separate and heuristic manner. In this study, we develop a lifelong collaborative model (LCM) based on non-negative matrix factorization to accurately learn topics and domain-specific word embeddings. LCM particularly investigates:(1) developing a knowledge graph based on the semantic relationships among words in the lifelong learning process, so as to accumulate global context information discovered by topic models and local context information reflected by context word embeddings from previous domains, and (2) developing a subword graph based on byte pair encoding and pairwise word relationships to exploit subword information of words in the current in-domain corpus. To the best of our knowledge, we are the first to collaboratively learn topics and word embeddings via lifelong learning. Experiments on real-world in-domain text streams validate the effectiveness of our method.

show abstract

“…For completeness, we also replace the SVM classifier with a neural network classifier (Chen et al, 2020a) consisting of 3 fully-connected layers. The results are shown in Appendix I.…”

Section: Word Embedding Learningmentioning

confidence: 99%

Lifelong Learning of Topics and Domain-Specific Word Embeddings

Qin¹,

Lu²,

Rao³

2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

show abstract

“…However, from the perspective of symbolic evolution, Chinese symbols have always possessed their unique structural features and peculiarities, developing from the initial hieroglyphics to their present forms as the Chinese characters shown in Figure 1. Recent researches [6,38] also demonstrate that the glyphs of Chinese characters contain rich semantic information and have the potential to enhance the word representation of them. Meng et al [26] first apply the glyph features of Chinese characters into the pre-trained model BERT and achieve significant performance on many Chinese NLU tasks, such as Named Entity Recognition [20], News Text Classification [22] and Sentiment Analysis [40].…”

Section: Related Workmentioning

confidence: 99%

“…Hence, the glyphs of Chinese characters can convey some meanings in many cases, and Chinese characters with similar structures can have intrinsic links. They Based on the above observations, some methods [6,26,38] incorporate the glyph features to enhance the Chinese character representation already covered into character embeddings (character ID-based), e.g., Meng et al [26] combine the glyph features extracted from various forms of Chinese characters with the BERT embeddings. They demonstrate that the glyph features of Chinese characters are authentically helpful to improve the performance of models.…”

Section: Introductionmentioning

confidence: 99%

GlyphCRM: Bidirectional Encoder Representation for Chinese Character with its Glyph

Li,

Zhao,

et al. 2021

Preprint

View full text Add to dashboard Cite

Previous works indicate that the glyph of Chinese characters contains rich semantic information and has the potential to enhance the representation of Chinese characters. The typical method to utilize the glyph features is by incorporating them into the character embedding space. Inspired by previous methods, we innovatively propose a Chinese pre-trained representation model named as GlyphCRM, which abandons the ID-based character embedding method yet solely based on sequential character images. We render each character into a binary grayscale image and design two-channel position feature maps for it. Formally, we first design a two-layer residual convolutional neural network, namely HanGlyph to generate the initial glyph representation of Chinese characters, and subsequently adopt multiple bidirectional encoder Transformer blocks as the superstructure to capture the contextsensitive information. Meanwhile, we feed the glyph features extracted from each layer of the HanGlyph module into the underlying Transformer blocks by skip-connection method to fully exploit the glyph features of Chinese characters. As the HanGlyph module can obtain a sufficient glyph representation of any Chinese character, the long-standing out-of-vocabulary problem could be effectively solved. Extensive experimental results indicate that GlyphCRM substantially outperforms the previous BERT-based state-of-theart model on 9 fine-tuning tasks, and it has strong transferability and generalization on specialized fields and low-resource tasks. We

show abstract

“…Several related works have been published to explore more feasibility. For instance, GWE directly extracted character glyph features from the bitmaps of characters by convolutional auto-encoder (ConvAE) [27], Glyph2Vec extracted visual features from word glyphs to expand current word embedding space for out-of-vocabulary word embedding, without the need of accessing any corpus [28], and Glyce designed Convolutional Neural Network (CNN) structures (called tianzige-CNN) tailored to Chinese character image processing [29]. These models extract the morphological feature by traditional image processing methods instead of linguistic knowledge, which introduces the connection of characters, and the noise from the unclarity of image processing algorithms.…”

Section: Introductionmentioning

confidence: 99%

“…The results of the word similarity task. ± 0.33 59 28. ± 0.21 58.08 ± 0.32 31.84 ± 0.24 50.77 ± 0.05 57.32 ± 0.28 38.35 ± 0.27 31.93 ± 0.05 Skipgram 58.32 ± 0.40 60.07 ± 0.21 49.18 ± 0.05 33.18 ± 0.22 56.42 ± 0.27 61.6 ± 0.03 31.66 ± 0.11 31.79 ± 0.32 CWE 57.58 ± 0.34 59.46 ± 0.18 50.70 ± 0.01 33.98 ± 0.49 56.76 ± 0.01 62.18 ± 0.21 33.91 ± 0.48 30.84 ± 0.21 JWE 52.04 ± 0.21 62.51 ± 0.68 57.46 ± 0.54 35.06 ± 0.58 54.83 ± 0.15 66.05 ±0.40 43.95 ± 0.09 36.97 ± 0.04 cw2vec 53.81 ± 0.03 57.54 ± 0.30 57.96 ± 0.29 23.88 ± 0.21 56.32 ± 0.07 61.29 ± 0.85 37.61 ± 0.49 27.66 ± 0.18 RECWE 53.73 ± 0.22 61.71 ± 0.14 32.40 ± 0.14 32.24 ± 0.07 53.73 ± 0.22 62.21 ± 0.36 54.4 ± 0.14 32.24 ± 0.07 4CWE 53.18 ± 0.05 61.61 ± 0.46 63.30 ± 0.20 37.47 ± 0.34 53.14 ± 0.21 63.54 ± 0.65 50.38 ± 0.60 39.56 ± 0.14 4CWE(ATW) 53.26 ± 0.30 62.04 ± 0.04 63.36 ±0.25 37.11 ± 0.02 53.19 ± 0.48 63.98 ± 0.33 51.51 ±0.52 39.43 ± 0.03 4CWE(ACW) 53.70 ± 0.38 62.58 ±0.06 60.52 ± 0.94 38.44 ±0.35 53.79 ± 0.15 62.91 ± 0.52 48.82 ± 0.12 40.02 ±0.05…”

mentioning

confidence: 97%