Normalização textual de conteúdo gerado por usuário

Abstract

User Generated Content (UGC) is the name given to content created spontaneously by ordinary individuals, without connections to the media. This type of content carries valuable information and can be exploited by several areas of knowledge. Much of the UGC is provided in the form of texts -product reviews, comments on forums about movies, and discussions on social networks are examples. However, the language used in UGC texts differs, in many ways, from the cultured norm of the language, making it difficult for NLP techniques to handle them. UGC language is strongly linked to the language used in daily life, containing a large amount of noise. Spelling mistakes, abbreviations, slang, absence or misuse of punctuation and capitalization are some noises that make it difficult to process these texts. Several works report considerable loss of performance when testing NLP state-of-the-art tools in UGC texts. Textual Normalization is the process of turning noisy words into words considered correct and can be used to improve the quality of UGC texts. This work reports the development of methods and systems that aim to (a) identify noisy words in UGC, (b) find candidate words for substitution, and (c) rank candidates for normalization. For the identification of noisy words, lexical-based methods and machine learning ones using deep neural networks were proposed. The automatic identification presented results comparable to the use of lexicons, proving that this process can be done with low dependence of resources. For the generation and ranking of candidates, techniques based on lexical similarity and word embeddings were investigated. It was concluded that the use of embeddings is highly suitable for normalization, having achieved the best results. All proposed methods were evaluated based on a UGC corpus annotated throughout the project, containing texts from different sources: discussion forums, product reviews and tweets. A system, Enelvo, combining all methods was implemented and compared to another existing normalizing system, UGCNormal. The results obtained by the Enelvo system were considerably higher, with a correction rate between 67 % and 97 % for different types of noise, with less dependence on resources and greater flexibility in normalization.

Tipologia de fenômenos ortográficos e lexicais em CGU: o caso dos tweets do mercado financeiro

“…Sobre os tweets e gêneros similares, Bertaglia (2017), por exemplo, visando à construção de ferramentas de normalização para UGC, investigou um corpus em português composto por tweets, postagens de um fórum de discussão e análises de produtos. O autor identificou 3.699 palavras distintas que não constavam em um dicionário de referência e anotou essas palavras em função de 8 categorias de desvios da língua padrão: (i) erro ortográfico (e de digitação), (ii) acrônimo, (iii) abreviação, (iv) internetês, (v) estrangeirismo, (vi) unidade de medida, (viii) nome próprio, e (vii) sem categoria (isto é, tokens cuja classificação não é clara ou varia conforme o contexto).…”

Tecnologias da Linguagem, Ética em IA e Regulamentação

“…É preciso enfatizar que os danos causados por tais tecnologias variam em escala e natureza. Por um lado, ao naturalizar uma variante privilegiada enquanto norma padrão [1], tais tecnologias cometem violência simbólica ao classificarem variantes como "erradas" ou que "precisam passar por normalização" [4]. Por outro lado, podem também produzir violência material ao mascarar processos de tomada de decisão inerentemente injustos sob uma ilusão de objetividade técnica.…”

A context‐aware recommender method based on text and opinion mining

“…Reviews' texts are sources of large amounts of noise, mainly due to their informal nature. Thus, it is natural to consider normalization to improve the quality of reviews' texts (Bertaglia, 2017).…”