Lexical Correction of Polish Twitter Political Data

Abstract

Language processing architectures are often evaluated in near-to-perfect conditions with respect to processed content. The tools which perform sufficiently well on electronic press, books and other type of non-interactive content may poorly handle noisy, colloquial and multilingual textual data which make the majority of communication today. This paper aims at investigating how Polish Twitter data (in a slightly controlled 'political' flavour) differs from expectation of linguistic tools and how it could be corrected to be ready for processing by standard language processing chains available for Polish. The setting includes specialised components for spelling correction of tweets as well as hashtag and username decoding.

Evaluation of basic modules for isolated spelling error correction in Polish texts

“…Another simple approach is the aforementioned diacritical swapping, which is a term that we introduce here for referring to a solution inspired by the work of (Ogrodniczuk and Kopeć, 2017). Namely, from the incorrect form we try to produce all strings obtainable by either adding or removing diacritical marks from characters.…”

“…Error cases provided by PlEWi are, therefore, not a balanced representation of spelling errors in written Polish language. PlEWi does have the advantage of scale in comparison to existing literature, such as (Ogrodniczuk and Kopeć, 2017) operating on a set of only 740 annotated errors in tweets.…”

“…Spelling correction tests described in literature have tended to focus on one approach applied to a specific corpus. Limited examples include works on spellchecking mammography reports and tweets (Mykowiecka and Marciniak, 2007;Ogrodniczuk and Kopeć, 2017). These works emphasized the importance of tailoring correction systems to specific problems of corpora they are applied to.…”

“…When the user forgets or neglects to press the AltGr key, typos such as writing *olowek instead of ołówek appear. In fact, (Ogrodniczuk and Kopeć, 2017) managed to get substantial performance on Twitter corpus by using this "diacritical swapping" alone.…”

“…Similar considerations eliminated from our focus handcrafted solutions for the whole spelling correction pipeline, primarily the LanguageTool (Miłkowski, 2010). Its performance in fixing spelling of Polish tweets was already tested (Ogrodniczuk and Kopeć, 2017). For our purposes it would be given an unfair advantage, since it is a rulebased system making heavy use of words in context of the error.…”