Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation

Heinzerling, Benjamin; Strube, Michael

doi:10.18653/v1/p19-1027

Cited by 39 publications

(36 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When task-specific supervision is limited, e.g. sequence labeling in low resource languages, mBERT performs better than fastText while underperforming a single BPEmb trained on all languages (Heinzerling and Strube, 2019). Contrary to this work, we focus on mBERT from the perspective of representation learning for each language in terms of monolingual corpora resources and analyze how to improve BERT for low resource languages.…”

Section: Representations For Low Resource Languagesmentioning

confidence: 99%

Are All Languages Created Equal in Multilingual BERT?

Wu¹,

Dredze²

2020

Proceedings of the 5th Workshop on Representation Learning for NLP

164

109

View full text Add to dashboard Cite

Multilingual BERT (mBERT) (Devlin, 2018) trained on 104 languages has shown surprisingly good cross-lingual performance on several NLP tasks, even without explicit crosslingual signals (Wu and Dredze, 2019;Pires et al., 2019). However, these evaluations have focused on cross-lingual transfer with highresource languages, covering only a third of the languages covered by mBERT. We explore how mBERT performs on a much wider set of languages, focusing on the quality of representation for low-resource languages, measured by within-language performance. We consider three tasks: Named Entity Recognition (99 languages), Part-of-speech Tagging, and Dependency Parsing (54 languages each). mBERT does better than or comparable to baselines on high resource languages but does much worse for low resource languages. Furthermore, monolingual BERT models for these languages do even worse. Paired with similar languages, the performance gap between monolingual BERT and mBERT can be narrowed. We find that better models for low resource languages require more efficient pretraining techniques or more data.

show abstract

Section: Representations For Low Resource Languagesmentioning

confidence: 99%

Are All Languages Created Equal in Multilingual BERT?

Wu¹,

Dredze²

2020

Proceedings of the 5th Workshop on Representation Learning for NLP

164

109

View full text Add to dashboard Cite

show abstract

“…This is the baseline setting. We also compare our results to multilingual embeddings (Multi) which have been successfully used in monolingual settings as well (Heinzerling and Strube, 2019). To ensure comparability, we use the multilingual versions of BPEmb and FLAIR, which were trained simultaneously on 275 and 300 languages, respectively.…”

Section: Experiments and Resultsmentioning

confidence: 99%

On the Choice of Auxiliary Languages for Improved Sequence Tagging

Lange

Adel

Strötgen

2020

Proceedings of the 5th Workshop on Representation Learning for NLP

View full text Add to dashboard Cite

Recent work showed that embeddings from related languages can improve the performance of sequence tagging, even for monolingual models. In this analysis paper, we investigate whether the best auxiliary language can be predicted based on language distances and show that the most related language is not always the best auxiliary language. Further, we show that attention-based meta-embeddings can effectively combine pre-trained embeddings from different languages for sequence tagging and set new state-of-the-art results for part-of-speech tagging in five languages.

show abstract

“…Furthermore, [79] found that although mBERT performs well in scenarios with medium and high language resources, non-contextual embedding working at the sub-word level, such as BPEmb, outperforms mBERT in low-resources scenarios. Finally, [80] explored cross-lingual transfer for Danish using several architectures for supervised NER, including Flair, fastText, BPE and both monolingual (Danish) and multilingual BERT, on a modestly-sized training set, testing different training and fine-tuning approaches.…”

Section: Background and Related Workmentioning

confidence: 99%

Crosslingual named entity recognition for clinical de-identification applied to a COVID-19 Italian data set

Catelli

Gargiulo

Casola

et al. 2020

Applied Soft Computing

View full text Add to dashboard Cite

The COrona VIrus Disease 19 (COVID-19) pandemic required the work of all global experts to tackle it. Despite the abundance of new studies, privacy laws prevent their dissemination for medical investigations: through clinical de-identification, the Protected Health Information (PHI) contained therein can be anonymized so that medical records can be shared and published. The automation of clinical de-identification through deep learning techniques has proven to be less effective for languages other than English due to the scarcity of data sets. Hence a new Italian de-identification data set has been created from the COVID-19 clinical records made available by the Italian Society of Radiology (SIRM). Therefore, two multi-lingual deep learning systems have been developed for this low-resource language scenario: the objective is to investigate their ability to transfer knowledge between different languages while maintaining the necessary features to correctly perform the Named Entity Recognition task for de-identification. The systems were trained using four different strategies, using both the English Informatics for Integrating Biology & the Bedside (i2b2) 2014 and the new Italian SIRM COVID-19 data sets, then evaluated on the latter. These approaches have demonstrated the effectiveness of cross-lingual transfer learning to de-identify medical records written in a low resource language such as Italian, using one with high resources such as English.

show abstract

Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation

Cited by 39 publications

References 32 publications

Are All Languages Created Equal in Multilingual BERT?

Are All Languages Created Equal in Multilingual BERT?

On the Choice of Auxiliary Languages for Improved Sequence Tagging

Crosslingual named entity recognition for clinical de-identification applied to a COVID-19 Italian data set

Contact Info

Product

Resources

About