Word-level language identification in<i>The Chymistry of Isaac Newton</i>

King, Levi; Kübler, Sandra; Hooper, Wallace

doi:10.1093/llc/fqu032

Cited by 9 publications

(9 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mukherjee et al (2014) used language labels of surrounding words with NB. King et al (2015) used the language probabilities of the previous word to determining weights for languages. King et al (2014b) used unigram, bigram and trigram language label transition probabilities.…”

Section: Statistics Of Words Van Der Lee and Boschmentioning

confidence: 99%

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

104

View full text Add to dashboard Cite

Language identification ("LI") is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used in the LI literature. We describe the features and methods using a unified notation, to make the relationships between methods clearer. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.LI as a task predates computational methods -the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional LI program for text is by Mustonen (1965), a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible.In the early 1970s, Nakamura (1971) considered the problem of automatic LI. According to Rau (1974) and the available abstract of Nakamura's article, 1 his language identifier was able to distinguish between 25 languages written with the Latin alphabet. As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual LI. Rau (1974) wrote his master's thesis "Language Identification by Statistical Analysis" for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use LI of text in military intelligence settings is evidenced by the recent articles of, for example, Rafidha Rehiman et al. (2013), Rowe et al. (2013), and Voss et al. (2014. As features for LI, Rau (1974) used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic (K), he managed...

show abstract

Section: Statistics Of Words Van Der Lee and Boschmentioning

confidence: 99%

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

104

View full text Add to dashboard Cite

show abstract

“…They occur in all digital techniques that access digitized sources through words. Thus, virtually every digital method for working with textual sources is confronted with these problems, not just retrieval methods such as source selection through keyword search, but also more quantitative analytical methods (e.g., King, K€ ubler, and Hooper 2015;Lijffijt et al 2016;Verhoef 2015, 56-57). As soon as one uses words to access sources, one is confronted with the equivocality of natural language: The meaning of words varies between contexts.…”

Section: Assessing Digital Techniquesmentioning

confidence: 99%

Phrasing history: Selecting sources in digital repositories

Huistra

Mellink

2016

Historical Methods: A Journal of Quantitative and Interdiscipli

View full text Add to dashboard Cite

In recent years, mass digitization has opened up voluminous text corpora to human interpretation. Full-text search lets historians now find new sources that can change their understanding of thoroughly studied historical episodes. At the same time, it forces scholars to access historical sources in a new way: through specific words. This article analyses the consequences of this new way of accessing sources and investigates which search technologies are best suited for historical source selection in digital repositories. It argues that to seize the opportunities that digitization offers, historians must refine their search technologies so that they are based on words but are less dependent on exact phraseology.

show abstract

“…Several models for MAN-EN codeswitched language identification were developed as part of the First Shared Task on Language Identification in Codeswitched Data King et al, 2014). The most common technique was to employ supervised machine learning algorithms (e.g., extended Markov Models and Conditional Random Field) to train a classifier.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the Second Workshop on Computational Approaches to Code Switching

2016

View full text Add to dashboard Cite

ii Introduction Code-switching (CS) is the phenomenon by which multilingual speakers switch back and forth between their common languages in written or spoken communication. CS is pervasive in informal text communications such as news groups, tweets, blogs, and other social media of multilingual communities. Such genres are increasingly being studied as rich sources of social, commercial and political information. Apart from the informal genre challenge associated with such data within a single language processing scenario, the CS phenomenon adds another significant layer of complexity to the processing of the data. Efficiently and robustly processing CS data presents a new frontier for our NLP algorithms on all levels. The goal of this workshop is to bring together researchers interested in exploring these new frontiers, discussing state of the art research in CS, and identifying the next steps in this fascinating research area.The workshop program includes exciting papers discussing new approaches for CS data and the development of linguistic resources needed to process and study CS. We received a total of 12 regular workshop submissions of which we accepted nine for publication four of them as workshop talks and five as posters. The accepted workshop submissions cover a wide variety of language combinations from languages such as English, Hindi, Swahili, Mandarin, Dialectical Arabic and Modern Standard Arabic. The majority of the papers focus on social media data such as Twitter, and discussion fora.Another component of the workshop is the Second Shared Task on Language Identification of CS Data. The shared task focused on social media and included two language pairs: Modern Standard ArabicDialectal Arabic and English-Spanish. We received a total of 14 system runs from nine different teams. All teams except one submitted a shared task paper describing their system. All shared task systems will be presented during the workshop poster session and two of them will also present a talk. We would like to thank all authors who submitted their contributions to this workshop and all shared task participants for taking on the challenge of language identification in code switched data. We also thank the program committee members for their help in providing meaningful reviews. Lastly, we thank the EMNLP 2016 organizers for the opportunity to put together this workshop. AbstractThis paper addresses challenges of Natural Language Processing (NLP) on non-canonical multilingual data in which two or more languages are mixed. It refers to code-switching which has become more popular in our daily life and therefore obtains an increasing amount of attention from the research community. We report our experience that covers not only core NLP tasks such as normalisation, language identification, language modelling, part-of-speech tagging and dependency parsing but also more downstream ones such as machine translation and automatic speech recognition. We highlight and discuss the key problems for each of the tasks with supporting...

show abstract

Word-level language identification inThe Chymistry of Isaac Newton

Cited by 9 publications

References 8 publications

Automatic Language Identification in Texts: A Survey

Automatic Language Identification in Texts: A Survey

Phrasing history: Selecting sources in digital repositories

Proceedings of the Second Workshop on Computational Approaches to Code Switching

Contact Info

Product

Resources

About