Improved N-grams Approach for Web Page Language Identification

Selamat, Ali

doi:10.1007/978-3-642-24016-4_1

Cited by 9 publications

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cavnar et al (1994) were probably the first; they used the 300 most frequent character n-grams (with n ranging from 1 to 5, as is also typically used in other works). All the n-gram-based ap-proaches differ primarily in the calculation of the distance between the n-gram profile of the training and test text (Selamat, 2011;Yang and Liang, 2010), or by using additional features on top of the n-gram profiles (Padma et al, 2009;Carter et al, 2013). One of the fairly robust definitions of the distance (or similarity) was proposed by Choong et al (2009) who simply check the proportion of n-gram types seen in the tested document of the most frequent n-gram types extracted from training documents for each language.…”

Section: Related Workmentioning

confidence: 99%

LanideNN: Multilingual Language Identification on Text Stream

Kocmi

Bojar

2017

Proceedings of the 15th Conference of the European Chapter of The Association for Computational Linguistics: Volume 1

View full text Add to dashboard Cite

In language identification, a common first step in natural language processing, we want to automatically determine the language of some input text. Monolingual language identification assumes that the given document is written in one language. In multilingual language identification, the document is usually in two or three languages and we just want their names. We aim one step further and propose a method for textual language identification where languages can change arbitrarily and the goal is to identify the spans of each of the languages.Our method is based on Bidirectional Recurrent Neural Networks and it performs well in monolingual and multilingual language identification tasks on six datasets covering 131 languages. The method keeps the accuracy also for short documents and across domains, so it is ideal for off-the-shelf use without preparation of training data.

show abstract

Section: Related Workmentioning

confidence: 99%

LanideNN: Multilingual Language Identification on Text Stream

Kocmi

Bojar

2017

Proceedings of the 15th Conference of the European Chapter of The Association for Computational Linguistics: Volume 1

View full text Add to dashboard Cite

show abstract

“…Automatic language identification of text documents was addressed early on as an information processing task (Beesley 1988; Singh 2006). Further approaches to analysing web page language is also available (Selamat 2011). How to assess quality on the level of punctuation and misspelling and approaches to developing automated grading systems are discussed in Agichtein et al (2008).…”

Section: Conclusion and Taking It Forwardmentioning

confidence: 99%

Closing the loop: Assisting archival appraisal and information retrieval in one sweep

Kim

Ross

2013

Proc of Assoc for Info

View full text Add to dashboard Cite

In this article, we examine the similarities between the concept of appraisal, a process that takes place within the archives, and the concept of relevance judgement, a process fundamental to information retrieval systems. More specifically, we revisit appraisal/selection criteria proposed as a result of archival and digital curation communities, and, compare them to relevance criteria as discussed within information retrieval's literature based discovery. We illustrate how closely these criteria relate to each other and discuss how understanding the relationships between the these disciplines could form a basis for proposing automated appraisal and selection for archival processes and enabling complex queries within information retrieval.

show abstract

Text Language Identification Using Attention-Based Recurrent Neural Networks

Perełkiewicz

Poświata

2019

Artificial Intelligence and Soft Computing

View full text Add to dashboard Cite

Improved N-grams Approach for Web Page Language Identification

Cited by 9 publications

References 21 publications

LanideNN: Multilingual Language Identification on Text Stream

LanideNN: Multilingual Language Identification on Text Stream

Closing the loop: Assisting archival appraisal and information retrieval in one sweep

Text Language Identification Using Attention-Based Recurrent Neural Networks

Contact Info

Product

Resources

About