Predicting the Performance of Multilingual NLP Models

Abstract

Recent advancements in NLP have given us models like mBERT and XLMR that can serve over 100 languages. The languages that these models are evaluated on, however, are very few in number, and it is unlikely that evaluation datasets will cover all the languages that these models support. Potential solutions to the costly problem of dataset creation are to translate datasets to new languages or use template-filling based techniques for creation. This paper proposes an alternate solution for evaluating a model across languages which make use of the existing performance scores of the model on languages that a particular task has test sets for. We train a predictor on these performance scores and use this predictor to predict the model's performance in different evaluation settings. Our results show that our method is effective in filling the gaps in the evaluation for an existing set of languages, but might require additional improvements if we want it to generalize to unseen languages.

On the Economics of Multilingual Few-shot Learning: Modeling the Cost-Performance Trade-offs of Machine Translated and Manual Data

“…However, in practice, one would like to understand the trade-offs before collecting the data. Recently, Srinivasan et al (2021) showed that it is possible to predict the zero-shot and few-shot performance of MMLMs for different languages using linguistic properties and their representation in the pre-training corpus. Understanding if there exists a similar dependence of the performance trade-offs with the linguistic properties of different languages can help us generalize our framework to the new languages without the need for explicit data collection.…”

Beyond Static models and test sets: Benchmarking the potential of pre-trained models across tasks and languages

“…We consider two different regression models to estimate the perfor-mance in our experiments. i) XGBoost: We use the popular Tree Boosting algorithm XGBoost for solving the regression problem, which has been previously shown to achieve impressive results on the task (Xia et al, 2020;Srinivasan et al, 2021).…”

“…Xia et al (2020) showed that it is possible to build regression models that can accurately predict evaluation scores of NLP models under different experimental settings using various linguistic and dataset specific features. Srinivasan et al (2021) (c) Number of multilingual tasks containing test data for each of the 106 languages supported by the MMLMs (mBERT, XLMR). The bars are shaded according to the class taxonomy proposed by Joshi et al (2020).…”

“…Finally, there are a number of ways in which the current performance prediction methods can be improved for a more reliable estimation. Both Xia et al (2020); Srinivasan et al (2021) observed that these models can struggle to generalize on languages or configurations that have features that are remarkably different from the training data. Multitask learning as hinted by Lin et al (2019) and our experiments with Group Lasso can be a possible way to address this issue.…”

Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models

“…Lauscher et al (2020) recently, showed that it is possible to predict the zero shot performance of mBERT and XLM-R on different languages by formulating it as a regression problem, with pretraining data size and typological similarities between the pivot and target languages as the input features, and the performance on downstream task as the prediction target. Along similar lines Srinivasan et al (2021) and Dolicki and Spanakis (2021) explore zero-shot performance prediction with a larger set of features and different regression techniques.…”