Investigation and modeling of the structure of texting language

Choudhury, Monojit; Saraf, Rahul; Jain, Vijit; Mukherjee, Animesh; Sarkar, Sudeshna; Basu, Anupam

doi:10.1007/s10032-007-0054-0

Cited by 142 publications

(98 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Many authors evaluate on private tweet collections and/or on the text message corpus of Choudhury et al (2007). For English, Han and Baldwin (2011) created a small tweet dataset annotated with normalized variants at the word level.…”

Section: Normalization Datasetsmentioning

confidence: 99%

Normalizing tweets with edit scripts and recurrent neural embeddings

Chrupała¹

2014

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

Tweets often contain a large proportion of abbreviations, alternative spellings, novel words and other non-canonical language. These features are problematic for standard language analysis tools and it can be desirable to convert them to canonical form. We propose a novel text normalization model based on learning edit operations from labeled data while incorporating features induced from unlabeled data via character-level neural text embeddings. The text embeddings are generated using an Simple Recurrent Network. We find that enriching the feature set with text embeddings substantially lowers word error rates on an English tweet normalization dataset. Our model improves on stateof-the-art with little training data and without any lexical resources.

show abstract

Section: Normalization Datasetsmentioning

confidence: 99%

Normalizing tweets with edit scripts and recurrent neural embeddings

Chrupała¹

2014

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

show abstract

“…Let T be a sequence of texting forms t 1 t 2 ...t n , which are the texting language realization of the standard forms, and may differ from the standard forms. Given a sequence of texting forms T , the challenge is then to determine the corresponding standard forms S. Following Choudhury et al (2007)-and various approaches to spelling error correction, such as, e.g., Mays et al (1991)-we model text message normalization using a noisy channel. We want to find argmax S P (S|T ).…”

Section: An Unsupervised Noisy Channel Model For Text Message Normalimentioning

confidence: 99%

An unsupervised model for text message normalization

Cook

Stevenson

2009

Proceedings of the Workshop on Computational Approaches to Linguistic Creativity - CALC '09

View full text Add to dashboard Cite

Cell phone text messaging users express themselves briefly and colloquially using a variety of creative forms. We analyze a sample of creative, non-standard text message word forms to determine frequent word formation processes in texting language. Drawing on these observations, we construct an unsupervised noisy-channel model for text message normalization. On a test set of 303 text message forms that differ from their standard form, our model achieves 59% accuracy, which is on par with the best supervised results reported on this dataset.

show abstract

“…Social text, which has been growing and evolving steadily, has its own lexical and grammatical features (Choudhury et al, 2007;Eisenstein, 2013).…”

Section: Introductionmentioning

confidence: 99%

A Graph-based Approach for Contextual Text Normalization

Sonmez

Özgür

2014

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

The informal nature of social media text renders it very difficult to be automatically processed by natural language processing tools. Text normalization, which corresponds to restoring the non-standard words to their canonical forms, provides a solution to this challenge. We introduce an unsupervised text normalization approach that utilizes not only lexical, but also contextual and grammatical features of social text. The contextual and grammatical features are extracted from a word association graph built by using a large unlabeled social media text corpus. The graph encodes the relative positions of the words with respect to each other, as well as their part-ofspeech tags. The lexical features are obtained by using the longest common subsequence ratio and edit distance measures to encode the surface similarity among words, and the double metaphone algorithm to represent the phonetic similarity. Unlike most of the recent approaches that are based on generating normalization dictionaries, the proposed approach performs normalization by considering the context of the non-standard words in the input text. Our results show that it achieves state-ofthe-art F-score performance on standard datasets. In addition, the system can be tuned to achieve very high precision without sacrificing much from recall.

show abstract

Investigation and modeling of the structure of texting language

Cited by 142 publications

References 23 publications

Normalizing tweets with edit scripts and recurrent neural embeddings

Normalizing tweets with edit scripts and recurrent neural embeddings

An unsupervised model for text message normalization

A Graph-based Approach for Contextual Text Normalization

Contact Info

Product

Resources

About