XenC: An Open-Source Tool for Data Selection in Natural Language Processing

Rousseau, Anthony

doi:10.2478/pralin-2013-0013

Cited by 45 publications

(41 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Czech monolingual corpus News-2016 was backtranslated to English using the single best system provided by the University of Edinburgh from WMT'16. 3 We then added five copies of Newscommentary and the news subcorpus from Czeng, as well as 5M sentences from the Czeng EU corpus randomly selected after running modified Moore-Lewis filtering with XenC (Rousseau, 2013). This resulted in about 14M parallel sentences.…”

Section: Data and Preprocessingmentioning

confidence: 99%

Limsi$@$wmt'17

Burlot

Safari

Labeau

et al. 2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

This paper describes LIMSI's submissions to the news shared task at WMT'17 for English into Czech and Latvian, as well as related experiments. This year's novelties consist in the use of a neural machine translation system with a factored output predicting simultaneously a lemma decorated with morphological information and a fine-grained part-of-speech. Such a type of system drew our attention to the specific step of reinflection, where lemmas and parts-of-speech are transformed into fully inflected words. Finally, we ran experiments showing an efficient strategy for parameter initialization, as well as data filtering procedures.

show abstract

Section: Data and Preprocessingmentioning

confidence: 99%

Limsi$@$wmt'17

Burlot

Safari

Labeau

et al. 2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

show abstract

“…In order to select the most appropriate amount of monolingual data, we employ data selection techniques based on cross-entropy criterion using 2 for that we used a modified version of the m2scorer script that could be distributed Xenc 3 (Rousseau, 2013). The selected data is determined in such a way that the corresponding LM minimize the perplexity calculated on the development set.…”

Section: Smt System For Error Correctionmentioning

confidence: 99%

Proceedings of the Second Workshop on Arabic Natural Language Processing

Ali¹,

Magdy²,

Renals³

2015

View full text Add to dashboard Cite

ii Foreword Assalamu 3alaykum wa nín hǎo! Welcome to the Second Arabic Natural Language Processing Workshop held at ACL 2015 in Beijing, China.A number of Arabic NLP (or Arabic NLP-related) workshops and conferences have taken place, both in the Arab World and in association with international conferences. The Arabic NLP workshop at ACL 2015 follows in the footsteps of these previous efforts to provide a forum for researchers to share and discuss their ongoing work. As in the first Arabic NLP workshop held at EMNLP 2014 in Doha, Qatar, this workshop includes a shared task on Automatic Arabic Error Correction, which was designed in the tradition of high profile NLP shared tasks such as CONLL's grammar/error detection and numerous machine translation campaigns by NIST/WMT/MEDAR, among others.We received 23 main workshop submissions and selected 15 (65%) for presentation in the workshop. Nine papers will be presented orally and six as part of a poster session. The presentation mode is independent of of the ranking of the papers. The papers cover a diverse set of topics from designing orthography conventions and annotation tools to speech recognition and deep learning for sentiment analysis.The shared task was a success with eight teams from six countries participating. The shared task system descriptions (short) papers are included in the proceedings to document the shared task systems, but were not reviewed with the rest of the papers of the main workshop. These papers will be presented as posters. A long paper describing the shared task will be presented orally.The quantity and quality of the contributions to the main workshop, as well as the shared task, are strong indicators that there is a continued need for this kind of dedicated Arabic NLP workshop.We would like to acknowledge all the hard work of the submitting authors and thank the reviewers for their diligent work and for the valuable feedback they provided. We are also thankful to the work of the shared task committee, website committee and the publication co-chairs. It has been an honor to serve as program co-chairs. We hope that the reader of these proceedings will find them stimulating and beneficial. AbstractDifferent names may be popular in different countries. Hence, person names may give a clue to a person's country of origin. Along with other features, mapping names to countries can be helpful in a variety of applications such as country tagging twitter users. This paper describes the collection of Arabic Twitter user names that are either written in Arabic or transliterated into Latin characters along with their stated geographical locations. To classify previously unseen names, we trained naive Bayes and Support Vector Machine (SVM) multi-class classifiers using primarily bag-of-words features. We are able to map Arabic user names to specific Arab countries with 79% accuracy and to specific regions (Gulf, Egypt, Levant, Maghreb, and others) with 94% accuracy. As for transliterated Arabic names, the accuracy per country and per region was 67...

show abstract

“…We added five copies 5 of News-commentary and fully infl. the news subcorpus from Czeng, as well as 5M sentences from the Czeng EU corpus randomly selected after running modified Moore-Lewis filtering with XenC (Rousseau, 2013). The English-to-Latvian systems used all the parallel data provided at WMT'17.…”

Section: Data and Preprocessingmentioning

confidence: 99%

Word Representations in Factored Neural Machine Translation

Burlot¹,

García-Martínez²,

Barrault³

et al. 2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

Translation into a morphologically rich language requires a large output vocabulary to model various morphological phenomena, which is a challenge for neural machine translation architectures. To address this issue, the present paper investigates the impact of having two output factors with a system able to generate separately two distinct representations of the target words. Within this framework, we investigate several word representations that correspond to different distributions of morpho-syntactic information across both factors. We report experiments for translation from English into two morphologically rich languages, Czech and Latvian, and show the importance of explicitly modeling target morphology.

show abstract

XenC: An Open-Source Tool for Data Selection in Natural Language Processing

Cited by 45 publications

References 4 publications

Limsi$@$wmt'17

Limsi$@$wmt'17

Proceedings of the Second Workshop on Arabic Natural Language Processing

Word Representations in Factored Neural Machine Translation

Contact Info

Product

Resources

About