A Normalizer for UGC in Brazilian Portuguese

Abstract

User-generated contents (UGC) represent an important source of information for governments, companies, political candidates and consumers. However, most of the Natural Language Processing tools and techniques are developed from and for texts of standard language, and UGC is a type of text especially full of creativity and idiosyncrasies, which represents noise for NLP purposes. This paper presents UGCNormal, a lexicon-based tool for UGC normalization. It encompasses a tokenizer, a sentence segmentation tool, a phonetic-based speller and some lexicons, which were originated from a deep analysis of a corpus of product reviews in Brazilian Portuguese. The normalizer was evaluated in two different data sets and carried out from 31% to 89% of the appropriate corrections, depending on the type of text noise. The use of UGCNormal was also validated in a task of POS tagging, which improved from 91.35% to 93.15% in accuracy and in a task of opinion classification, which improved the average of F1-score measures (F1-score positive and F1-score negative) from 0.736 to 0.758.

Social media text normalization for Turkish

“…Qian et al 2015 use a transition-based model for joint segmentation, POS-tagging and normalization for the Chinese language. Duran et al 2015 propose a lexicon-based tool for user-generated content (UGC) normalization in Brazilian Portuguese. In recent years, we observe many studies focusing on social media text normalization for Spanish, which is a morphologically more interesting language (also carrying some similar challenges to Turkish such as the importance of diacritic restoration in social media text normalization).…”

Word Search with Trending Reviews on Twitter

“…Text normalization refers to the process of modifying the format of a text to communicate ideas for a particular purpose. [12,13]. Text normalization is crucial to employ since a text or reading will often contain words that are challenging to grasp, ranging from vocabulary used differently in different languages or contexts to sentences that have distinct meanings from their original context [14,15].…”