A Mixed Method Lemmatization Algorithm Using a Hierarchy of Linguistic Identities (HOLI)

Ingason, Anton Karl; Helgadóttir, Sigrún; Loftsson, Hrafn; Rögnvaldsson, Eiríkur

doi:10.1007/978-3-540-85287-2_20

Cited by 35 publications

(16 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Stemming is a simple experimental process which is often used to chops off the affixes of words to get the concrete and precise sense of the word. To obtain information using morphological techniques Stemming has been widely adopted for morphological technique for extracting information [13]. We have used five different databases for stemming our Korean text files, out of five databases 3 were individually built by our group members.…”

Section: Stemmingmentioning

confidence: 99%

Text Mining: Extraction of Interesting Association Rule with Frequent Itemsets Mining for Korean Language from Unstructured Data

Khan

Woo²,

Seo

et al. 2015

IJMUE

View full text Add to dashboard Cite

Text mining is a specific method to extract knowledge from structured and unstructured data. This extracted knowledge from text mining process can be used for further usage and discovery. This paper presents the method for extraction information from unstructured text data and the importance of Association Rules Mining, specifically for of Korean language (text) and also, NLP (Natural Language Processing) tools are explained. Association Rules Mining (ARM) can also be used for mining association between itemsets from unstructured data with some modifications. Which can then, help for generating statistical thesaurus, to mine grammatical rules and to search large data efficiently. Although various association rules mining techniques have successfully used for market basket analysis but very few has applied on Korean text. A proposed Korean language mining method calculates and extracts meaningful patterns (association rules) between words and presents the hidden knowledge. First it cleans and integrates data, select relevant data then transform into transactional database. Then data mining techniques are used on data source to extract hidden patterns. These patterns are evaluated by specific rules until we get the valid and satisfactory result. We have tested on Korean news corpus and results have shown that it has worked well, and the results were adequate enough to research further.These processes are done until we get the required result. Technically, data mining is the process of finding correlations or patterns among dozens of fields in large relational databases. We have used same technique to extract meaningful association rules out of Korean text. First step is to pre-process data (getting rid of un-necessary data), then we transformed process data into transactional database. Grouping this transactional database with other hand-built databases we mined some interesting and useful patterns in given Korean text data collection (corpus).Rest of the paper is structured as follows: Section 2 covers related work; Section 3 describes the association rules mining for Korean language. Experiments and results are in Sections 4, and Sections 5 are conclusion and future work.

show abstract

Section: Stemmingmentioning

confidence: 99%

Text Mining: Extraction of Interesting Association Rule with Frequent Itemsets Mining for Korean Language from Unstructured Data

Khan

Woo²,

Seo

et al. 2015

IJMUE

View full text Add to dashboard Cite

show abstract

“…Stemming programs are commonly referred to as stemming algorithms or stemmers. Ingason et al [14] attempt to convert a word to its linguistically correct root which ultimately facilitates the reduction of all words that possess an identical root to a single one. This is obtained by removing each word of its derivational and inflectional suffixes [4].…”

Section: Stemmingmentioning

confidence: 99%

Fingerprint-Based Near-Duplicate Document Detection with Applications to SNS Spam Detection

Kim

2014

International Journal of Distributed Sensor Networks

View full text Add to dashboard Cite

Social networking has been used widely by millions of people over the world. It has become the most popular way for people who want to connect and interact online with their friends. Currently, there are many social networking sites, for instance, Facebook, My Space, and Twitter, with a huge number of active users. Therefore, they are also good places for spammers or cheaters who want to steal the personal information of users or advertise their products. Recently, many proposed methods are applied to detect spam comments on social networks with different techniques. In this paper, we propose a similarity-based method that combines fingerprinting technique with trie-tree data structure and meet-in-the-middle approach in order to achieve a higher accuracy in spam comments detection. Using our proposed approach, we are able to detect around 98% spam comments in our dataset.

show abstract

“…The stems are used to represent the key terms of a query or document instead of the original word. Lemmatization [30] is an algorithm which attempts to convert a word to its linguistically correct root which ultimately facilitates the reduction of all words, possessing an identical root to a single one. This is obtained by removing each word of its derivational and inflectional suffixes [23].…”

Section: Stemmingmentioning

confidence: 99%

Near-Duplicate Web Page Detection: An Efficient Approach Using Clustering, Sentence Feature and Fingerprinting

Kumar¹,

Govindarajulu²

2013

IJCIS

View full text Add to dashboard Cite

Duplicate and near-duplicate web pages are the chief concerns for web search engines. In reality, they incur enormous space to store the indexes, ultimately slowing down and increasing the cost of serving results. A variety of techniques have been developed to identify pairs of web pages that are "similar" to each other. The problem of finding near-duplicate web pages has been a subject of research in the database and web-search communities for some years. In order to identify the near duplicate web pages, we make use of sentence level features along with fingerprinting method. When a large number of web documents are in consideration for the detection of web pages, then at first, we use K-mode clustering and subsequently sentence feature and fingerprint comparison is used. Using these steps, we exactly identify the near duplicate web pages in an efficient manner. The experimentation is carried out on the web page collections and the results ensured the efficiency of the proposed approach in detecting the near duplicate web pages.

show abstract

A Mixed Method Lemmatization Algorithm Using a Hierarchy of Linguistic Identities (HOLI)

Cited by 35 publications

References 7 publications

Text Mining: Extraction of Interesting Association Rule with Frequent Itemsets Mining for Korean Language from Unstructured Data

Text Mining: Extraction of Interesting Association Rule with Frequent Itemsets Mining for Korean Language from Unstructured Data

Fingerprint-Based Near-Duplicate Document Detection with Applications to SNS Spam Detection

Near-Duplicate Web Page Detection: An Efficient Approach Using Clustering, Sentence Feature and Fingerprinting

Contact Info

Product

Resources

About