On the unsupervised analysis of domain-specific Chinese texts

Abstract

With the growing availability of digitized text data both publicly and privately, there is a great need for effective computational tools to automatically extract information from texts. Because the Chinese language differs most significantly from alphabet-based languages in not specifying word boundaries, most existing Chinese text-mining methods require a prespecified vocabulary and/or a large relevant training corpus, which may not be available in some applications. We introduce an unsupervised method, top-down word discovery and segmentation (TopWORDS), for simultaneously discovering and segmenting words and phrases from large volumes of unstructured Chinese texts, and propose ways to order discovered words and conduct higher-level context analyses. TopWORDS is particularly useful for mining online and domain-specific texts where the underlying vocabulary is unknown or the texts of interest differ significantly from available training corpora. When outputs from TopWORDS are fed into context analysis tools such as topic modeling, word embedding, and association pattern finding, the results are as good as or better than that from using outputs of a supervised segmentation method.ue to the explosive growth of the Internet technology and the public adoption of the Internet as a main culture media, a large amount of text data is available. It is more and more attractive for many researchers to extract information from diverse text data to create new knowledge. Biomedical researchers can gain understanding on how diseases, symptoms, and other features are spatially, temporally, and ethnically distributed and associated with each other by mining research articles and electronic medical records. Marketers can learn what consumers say about their products and services by analyzing online reviews and comments. Social scientists can discover hot events from news articles, web pages, blogs, and tweets and infer driving forces behind them. Historians can extract information about historical figures from historical documents: who they were, what they did, and what social relationships they had with other historical figures.For alphabet-based languages such as English, many successful learning methods have been proposed (see ref. 1 for a review). For character-based languages such as Chinese and other East Asian languages, effective learning algorithms are still limited. Chinese has a much larger "alphabet" and vocabulary than English: Zhonghua Zihai Dictionary (2) lists 87,019 distinct Chinese characters, of which 3,000 are commonly used; and the vocabulary of Chinese is an open set when named entities are included. Additionally, morphological variations in Latin-derived languages (e.g., uppercase or lowercase letters, tense and voice changes), which provide useful hints for text mining, do not exist in Chinese. Because there is no space between Chinese characters in each sentence, significant ambiguities are present in deciphering its meaning.There are two critical challenges in processing Chinese texts: (i) word segment...

Text Mining National Commitments towards Agrobiodiversity Conservation and Use

“…Further research is needed to verify whether this result holds when a greater number of countries are included in the analysis. However, it potentially offers a powerful approach for rapidly assessing country commitments around the globe on a repeat-basis based on openly accessible data sources, provided language barriers in text-mining applications can be overcome [40] notably for Chinese characters [41]. Developing national-level, globally applicable measures such as these is critical for tracking national progress on SDG attainment and stimulating business buy-in [42].…”

Domain-Specific New Words Detection in Chinese

“…In this section, we propose a Domain Top-Words model. We introduce the Word Dictionary Model (Ge et al, 1999;Chang and Su, 1997;Cohen et al, 2007) and TopWords model proposed by Deng et al (2016) in subsection 3.1 and 3.2. Then we introduce our Domain TopWords model in subsection 3.3, 3.4 and 3.5.…”

“…These supervised models cannot be used in domain-specific words detection directly, due to the lack of annotated domain-specific data. In addition, there are also some unsupervised models, such as Top-Words proposed by Deng et al (2016). However, it needs time-consuming post-processing to extract the second type of domain-specific words.…”

“…These methods focus on extracting the first type of domain-specific words and conduct postprocessing to discover the second type of words. Deng et al proposed a statistical model Top-Words (Deng et al, 2016) to extract the first type of words, it can imply some of these statistical measures into the model itself. Besides, it designs a feature called relative frequency to extract the second type of domain-specific words.…”

Detecting new Chinese words from massive domain texts with word embedding

“…The frequent patterns are itemsets that appear in a dataset no less than a user-specified threshold. In recent years, frequent pattern mining has been introduced in NLP tasks as a necessary preprocessing method for extracting quality phrases from corpora [14,35]. In our work, the task of frequent pattern mining is conducted on the transaction dataset of T. Given the database T and the predefined threshold of minimum support = min s , the object of frequent pattern mining is to collect all the n-itemsets whose frequency is larger than min s into set FP (n) (T).…”