Automatic Term Extraction Based on Perplexity of Compound Words

Abstract

Abstract. Many methods of term extraction have been discussed in terms of their accuracy on huge corpora. However, when we try to apply various methods that derive from frequency to a small corpus, we may not be able to achieve sufficient accuracy because of the shortage of statistical information on frequency. This paper reports a new way of extracting terms that is tuned for a very small corpus. It focuses on the structure of compound terms and calculates perplexity on the term unit's left-side and right-side. The results of our experiments revealed that the accuracy with the proposed method was not that advantageous. However, experimentation with the method combining perplexity and frequency information obtained the highest average-precision in comparison with other methods.

Automatic Keyword Extraction for Wikification of East Asian Language Documents

“…For example, removing stop words using a stop word list [8], using part of speech tags [9], and using N-grams [10]. Yoshida and Nakagawa [11] proposed a keyword extraction method for Japanese language documents. This method uses perplexity on the term unit's left-side and right-side terms for extracting technical terms.…”

Care more about customers: Unsupervised domain-independent aspect detection for sentiment analysis of customer reviews

“…FLR is a word scoring method that uses internal structures and frequencies of candidates (FLR: Frequencies and Left and Right of the current word). One of the advantages of the FLR method is its size-robustness, that it can be applied to small corpus with less significant drop in performance than other standard methods like TF and IDF, because it is defined using more finegrained features [30].…”

Extracting Chinese multi-word terms from small corpus

“…The particularity of C-value is the consideration of the nested terms. Minoru [5] presented a method combining perplexity and frequency information to rank the candidates. Li [6] removed the non-term items from the candidates based on the CBC clustering algorithm.…”