Automated Concatenation of Embeddings for Structured Prediction

Wang, Xinyu; Jiang, Yong; Bach, Nguyen; Wang, Tao; Huang, Zhongqiang; Huang, Fei; Tu, Kewei

doi:10.18653/v1/2021.acl-long.206

Cited by 78 publications

(39 citation statements)

References 79 publications

(60 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Moreover, the embeddings are trained over long documents, which allows the model to easily model long-range dependencies to disambiguate complex named entities in the sentence. Recently, a lot of work shows that utilizing the document-level contexts in the CoNLL NER datasets can significantly improve token representations and achieves state-of-the-art performance (Yu et al, 2020;Luoma and Pyysalo, 2020;Yamada et al, 2020;Wang et al, 2021a). However, the lack of context in the MultiCoNER datasets means the embeddings cannot take advantage of long-range dependencies for entity disambiguation.…”

Section: Related Workmentioning

confidence: 99%

“…Shi and Lee (2021) proposed two-stage fine-tuning, which first trains a general multilingual Enhanced Universal Dependency (Bouma et al, 2021) parser and then finetunes on each specific language separately. Wang et al (2021a) proposed to train models through concatenating fine-tuned embeddings. We extend these ideas as multi-stage fine-tuning, which improves the accuracy of monolingual models that use finetuned multilingual embeddings as initialization in training.…”

Section: Related Workmentioning

confidence: 99%

“…ACE (Automated Concatenation of Embeddings) ACE (Wang et al, 2021a) is an improved version of CE which automatically selects a better concatenation of the embeddings. We use the same embedding types as CE and the knowledge are from our Wikipedia KB.…”

Section: Ce (Concatenation Of Embeddingsmentioning

confidence: 99%

See 2 more Smart Citations

DAMO-NLP at SemEval-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognition

Wang¹,

Shen²,

Cai³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The MultiCoNER shared task aims at detecting semantically ambiguous and complex named entities in short and low-context settings for multiple languages. The lack of contexts makes the recognition of ambiguous named entities challenging. To alleviate this issue, our team DAMO-NLP proposes a knowledge-based system, where we build a multilingual knowledge base based on Wikipedia to provide related context information to the named entity recognition (NER) model. Given an input sentence, our system effectively retrieves related contexts from the knowledge base. The original input sentences are then augmented with such context information, allowing significantly better contextualized token representations to be captured. Our system wins 10 out of 13 tracks in the Multi-CoNER shared task. 1 * : project lead. : equal contributions.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

DAMO-NLP at SemEval-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognition

Wang¹,

Shen²,

Cai³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…While named entity recognition in text has been studied extensively in the NLP community (Mikheev et al, 1999;Florian et al, 2003;Nadeau and Sekine, 2007;Ratinov and Roth, 2009;Ritter et al, 2011;Lample et al, 2016;Chiu and Nichols, 2016;Akbik et al, 2019;Wang et al, 2021b;Yamada et al, 2020), relatively little work has been conducted on extracting named entities from speech (Kim and Woodland, 2000;Sudoh et al, 2006;Parada et al, 2011;Caubrière et al, 2020;Yadav et al, 2020;Shon et al, 2021). Recognizing named entities from speech is a more challenging task which is commonly done through a pipeline approach: combining an automatic speech recognition (ASR) system with a text-based NER model (Sudoh et al, 2006;Raymond, 2013;Jannet et al, 2015).…”

Section: Spoken Named Entity Recognitionmentioning

confidence: 99%

On the Use of External Data for Spoken Named Entity Recognition

Pasad¹,

Wu²,

Shon³

et al. 2021

Preprint

View full text Add to dashboard Cite

Spoken language understanding (SLU) tasks involve mapping from speech audio signals to semantic labels. Given the complexity of such tasks, good performance might be expected to require large labeled datasets, which are difficult to collect for each new task and domain. However, recent advances in self-supervised speech representations have made it feasible to consider learning SLU models with limited labeled data. In this work we focus on low-resource spoken named entity recognition (NER) and address the question: Beyond selfsupervised pre-training, how can we use external speech and/or text data that are not annotated for the task? We draw on a variety of approaches, including self-training, knowledge distillation, and transfer learning, and consider their applicability to both end-to-end models and pipeline (speech recognition followed by text NER model) approaches. We find that several of these approaches improve performance in resource-constrained settings beyond the benefits from pre-trained representations alone. Compared to prior work, we find improved F1 scores of up to 16%. While the best baseline model is a pipeline approach, the best performance when using external data is ultimately achieved by an end-to-end model. We provide detailed comparisons and analyses, showing for example that end-to-end models are able to focus on the more NER-specific words.

show abstract

“…Standard Named Entity Recognition (NER) for high resource domains has already been successfully addressed [1,2]. In contrast, NER taggers often cannot achieve satisfying results in the historical domain.…”

Section: Introductionmentioning

confidence: 99%

hmBERT: Historical Multilingual Language Models for Named Entity Recognition

Schweter¹,

März²,

Schmid³

et al. 2022

Preprint

View full text Add to dashboard Cite

Compared to standard Named Entity Recognition (NER), identifying persons, locations, and organizations in historical texts forms a big challenge. To obtain machine-readable corpora, the historical text is usually scanned and optical character recognition (OCR) needs to be performed. As a result, the historical corpora contain errors. Also, entities like location or organization can change over time, which poses another challenge. Overall historical texts come with several peculiarities that differ greatly from modern texts and large labeled corpora for training a neural tagger are hardly available for this domain. In this work, we tackle NER for historical German, English, French, Swedish, and Finnish by training large historical language models. We circumvent the need for labeled data by using unlabeled data for pretraining a language model. hmBert, a historical multilingual BERT-based language model is proposed, with different sizes of it being publicly released. Furthermore, we evaluate the capability of hmBert by solving downstream NER as part of this year's HIPE-2022 shared task and provide detailed analysis and insights. For the Multilingual Classical Commentary coarse-grained NER challenge, our tagger HISTeria outperforms the other teams' models for two out of three languages.

show abstract

Automated Concatenation of Embeddings for Structured Prediction

Cited by 78 publications

References 79 publications

DAMO-NLP at SemEval-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognition

DAMO-NLP at SemEval-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognition

On the Use of External Data for Spoken Named Entity Recognition

hmBERT: Historical Multilingual Language Models for Named Entity Recognition

Contact Info

Product

Resources

About