German’s Next Language Model

Chan, Branden; Schweter, Stefan; Möller, Timo

doi:10.18653/v1/2020.coling-main.598

Cited by 108 publications

(98 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…BERT model for Twin network -For the BERT embedding block shown in Figure 1, we use a pretrained German BERT model, gbert-base (Chan et al, 2020;Devlin et al, 2018), fine-tuned on annotated data sampled from live traffic. For finetuning we use a dataset of 1.5 million live traffic utterances.…”

Section: Methodsmentioning

confidence: 99%

Combining semantic search and twin product classification for recognition of purchasable items in voice shopping

Le¹,

Weber²,

Bradford³

2021

Proceedings of the 4th Workshop on E-Commerce and NLP

View full text Add to dashboard Cite

The accuracy of an online shopping system via voice commands is particularly important and may have a great impact on customer trust. This paper focuses on the problem of detecting if an utterance contains actual and purchasable products, thus referring to a shopping-related intent in a typical Spoken Language Understanding architecture consisting of an intent classifier and a slot detector. Searching through billions of products to check if a detected slot is a purchasable item is prohibitively expensive. To overcome this problem, we present a framework that (1) uses a retrieval module that returns the most relevant products with respect to the detected slot, and (2) combines it with a twin network that decides if the detected slot is indeed a purchasable item or not. Through various experiments, we show that this architecture outperforms a typical slot detector approach, with a gain of +81% in accuracy and +41% in F1 score.

show abstract

Section: Methodsmentioning

confidence: 99%

Combining semantic search and twin product classification for recognition of purchasable items in voice shopping

Le¹,

Weber²,

Bradford³

2021

Proceedings of the 4th Workshop on E-Commerce and NLP

View full text Add to dashboard Cite

show abstract

“…For experiments with fine-tuning, we use language-specific BERT models 11 for German (Chan et al, 2020), Spanish (Canete et al, 2020), Dutch (de Vries et al, 2019, Finnish (Virtanen et al, 2019), Danish, 12 Croatain (Ulčar and Robnik-Šikonja, 2020), while we use mBERT (Devlin et al, 2019) for Afrikaans.…”

Section: Low Resource Settingmentioning

confidence: 99%

Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition

Ghaddar

Langlais

Ahmad

et al. 2021

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity. We introduce NRB, a new testbed carefully designed to diagnose Name Regularity Bias of NER models. Our results indicate that all state-of-the-art models we tested show such a bias; BERT fine-tuned models significantly outperforming feature-based (LSTM-CRF) ones on NRB, despite having comparable (sometimes lower) performance on standard benchmarks. To mitigate this bias, we propose a novel model-agnostic training method that adds learnable adversarial noise to some entity mentions, thus enforcing models to focus more strongly on the contextual signal, leading to significant gains on NRB. Combining it with two other training strategies, data augmentation and parameter freezing, leads to further gains.

show abstract

“…Regarding the classification, we fine-tune the pretrained gbert-base model (Chan et al, 2020), which has 110M parameters and is the best performing German transformer model for text classification at this number of parameters. While there is a larger gbert model available, we opted for the base variant due to its efficiency, which results in lower turnaround times of an AL step for the practitioner.…”

Section: Model and Trainingmentioning

confidence: 99%

Supporting Land Reuse of Former Open Pit Mining Sites using Text Classification and Active Learning

Schröder¹,

Annanias²,

Niekler³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Open pit mines left many regions worldwide inhospitable or uninhabitable. Many sites are left behind in a hazardous or contaminated state, show remnants of waste, or have other restrictions imposed upon them, e.g., for the protection of human or nature. Such information has to be permanently managed in order to reuse those areas in the future. In this work we present and evaluate an automated workflow for supporting the post-mining management of former lignite open pit mines in the eastern part of Germany, where prior to any planned land reuse, aforementioned information has to be acquired to ensure the safety and validity of such an endeavor. Usually, this information is found in expert reports, either in the form of paper documents, or in the best case as digitized unstructured text-all of them in German language. However, due to the size and complexity of these documents, any inquiry is tedious and time-consuming, thereby slowing down or even obstructing the reuse of related areas. Since no training data is available, we employ active learning in order to perform multi-label sentence classification for two categories of restrictions and seven categories of topics. The final system integrates optical character recognition (OCR), active-learningbased text classification, and geographic information system visualization in order to effectively extract, query, and visualize this information for any area of interest. Active learning and text classification results are twofold: Whereas the restriction categories were reasonably accurate (>0.85 F1), the seven topicoriented categories seemed to be complex even for human annotators and achieved mediocre evaluation scores (<0.70 F1).

show abstract

German’s Next Language Model

Cited by 108 publications

References 12 publications

Combining semantic search and twin product classification for recognition of purchasable items in voice shopping

Combining semantic search and twin product classification for recognition of purchasable items in voice shopping

Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition

Supporting Land Reuse of Former Open Pit Mining Sites using Text Classification and Active Learning

Contact Info

Product

Resources

About