A Graph-based Approach for Contextual Text Normalization

Abstract

The informal nature of social media text renders it very difficult to be automatically processed by natural language processing tools. Text normalization, which corresponds to restoring the non-standard words to their canonical forms, provides a solution to this challenge. We introduce an unsupervised text normalization approach that utilizes not only lexical, but also contextual and grammatical features of social text. The contextual and grammatical features are extracted from a word association graph built by using a large unlabeled social media text corpus. The graph encodes the relative positions of the words with respect to each other, as well as their part-ofspeech tags. The lexical features are obtained by using the longest common subsequence ratio and edit distance measures to encode the surface similarity among words, and the double metaphone algorithm to represent the phonetic similarity. Unlike most of the recent approaches that are based on generating normalization dictionaries, the proposed approach performs normalization by considering the context of the non-standard words in the input text. Our results show that it achieves state-ofthe-art F-score performance on standard datasets. In addition, the system can be tuned to achieve very high precision without sacrificing much from recall.

Combining the NER-OCR methods to improve information retrieval efficiency in the Indonesian posters

“…Kinerja NER-tools dapat ditingkatkan melalui normalisasi teks data uji dan menggunakan data latih dalam jumlah besar. Normalisasi dilakukan dengan melacak kata nonstandar untuk digantikan dengan kata standar yang sesuai [7], [17], sedangkan data latih dalam jumlah besar dapat diperoleh dengan mengolah artikel yang relevan dengan cakupan kasus [4], [9]- [11], [18], [19].…”

Untitled

“…However, the phonetic similarity used in these systems cannot be applied to Chinese words since Pinyin has its own specific characteristics, which do not easily map to English, for determining phonetic similarity. Another main application of phonetic similarity algorithms is text normalization (Xia et al, 2006;Li et al, 2003;Han et al, 2012;Sonmez and Ozgur, 2014;Qian et al, 2015), where phonetic similarity is measured by a combination of initial and final similarities. However, the encodings used in these approaches are too coarse-grained, yielding low F1 measures.…”

A Transition-based Model for Joint Segmentation, POS-tagging and Normalization

“…The task is generally treated as a noisy channel problem (Pennell and Liu, 2014;Cook and Stevenson, 2009;Yang and Eisenstein, 2013;Sonmez and Ozgur, 2014) or a translation problem (Aw et al, 2006;Contractor et al, 2010;Li and Liu, 2012;Zhang et al, 2014c). For English, most recent work (Han and Baldwin, 2011;Gouws et al, 2011;Han et al, 2012) uses two-step unsupervised approaches to first detect and then normalize informal words.…”