![](https://csdnimg.cn/release/download_crawler_static/12327607/bg4.jpg)
XTREME: A Benchmark for Evaluating Cross-lingual generalization
using cosine similarity.
4
Tatoeba
We use the Tatoeba dataset (Artetxe & Schwenk,
2019), which consists of up to 1,000 English-aligned sen-
tence pairs covering 122 languages. We find the nearest
neighbour using cosine similarity and calculate error rate.
3.3. Languages
As noted in Section 3.1, we choose our target languages
based on availability of monolingual data, and typological
diversity. We use the number of articles in Wikipedia as a
proxy for the amount of monolingual data available online.
In order to strike a balance between language diversity and
availability of monolingual data, we select all languages
out of the top 100 Wikipedias
5
with the most articles as of
December 2019.
6
We first select all languages that appear
in at least three of our benchmark datasets. This leaves
us with 19 languages, most of which are Indo-European
or major world languages. We now select 21 additional
languages that appear in at least one dataset and come from
less represented language families. Wherever possible, we
choose at least two languages per family.
7
In total, XTREME covers the following 40 languages (shown
with their ISO 639-1 codes for brevity) belonging to 12
language families and two isolates: af, ar, bg, bn, de, el, en,
es, et, eu, fa, fi, fr, he, hi, hu, id, it, ja, jv, ka, kk, ko, ml, mr,
ms, my, nl, pt, ru, sw, ta, te, th, tl, tr, ur, vi, yo, and zh. We
provide a detailed overview of these languages in terms of
their number of Wikipedia articles, linguistic features, and
coverage in XTREME in the appendix.
While XTREME covers these languages in the sense that
there is gold standard data in at least one task in each lan-
guage, this does not mean that it covers all aspects of each
language that are necessary for transfer. Languages may re-
veal different characteristics based on the task, domain, and
register in which they are used. XTREME thus only serves
as a glimpse into a model’s true cross-lingual generalization
capability.
3.4. Pseudo test data for analyses
XTREME covers 40 languages overall. Evaluation across the
majority of languages is only possible for a subset of tasks,
i.e. POS, NER, and Tatoeba. As additional diagnositics and
to enable a broader comparison across languages for a more
4
Results can be improved using more sophisticated similarity
metrics (Artetxe & Schwenk, 2019).
5
https://meta.wikimedia.org/wiki/List_of_
Wikipedias
6
This also has the benefit that they are covered by state-of-the-
art methods such as mBERT and XLM.
7
For the Austro-Asiatic, Kartvelian, and Kra-Dai families as
well as for isolates, we only obtain one language.
diverse set of tasks, we automatically translate the English
portions of a representative classification and QA task to the
remaining languages using an in-house translation system.
8
We choose XNLI and XQuAD as both have test sets that are
translations of the English data by professional translators.
We first verify that performance on the translated test sets
is a good proxy for performance on the gold standard test
sets. We report the detailed results in the appendix. For
XQuAD, the automatically translated test sets underesti-
mate mBERT’s true performance by 3.0 F1 / 0.2 EM points,
similar to the 2.6 F1 points reported by Agi
´
c & Schluter
(2018) when translating the test data to other languages.
9
For XNLI, the automatically translated test sets overestimate
the true prediction accuracy by 2.4 points. In order to mea-
sure the translation quality between the human-translated
test data and our pseudo test data, we compute the BLEU
score, and the chrF score (Popovi
´
c, 2015), which is suitable
for measuring the translation quality of some languages such
as Chinese and Russian. For the 14 languages in XNLI, we
obtain average scores of 34.2 BLEU and 58.9 chrF scores
on our pseudo test data compared to the reference transla-
tions, which correlate with a Pearson’s
ρ
of 0.57 and 0.28
respectively with mBERT performance.
Translating the English data to the remaining languages
yields 40-way parallel pseudo test data that we employ for
analyses in Section 5.
4. Experiments
4.1. Training and evaluation setup
XTREME focuses on the evaluation of multilingual represen-
tations. We do not place any restriction on the amount or
nature of the monolingual data used for pretraining multi-
lingual representations. However, we request authors to be
explicit about the data they use for training, in particular
any cross-lingual signal. In addition, we suggest authors
should not use any additional labelled data in the target task
beyond the one that is provided.
For evaluation, we focus on zero-shot cross-lingual trans-
fer with English as the source language as this is the most
common setting for the evaluation of multilingual represen-
tations and as many tasks only have training data available in
English. Although English is not generally the best source
language for cross-lingual transfer for all target languages
(Lin et al., 2019), this is still the most practically useful
setting. A single source language also facilitates evaluation
as models only need to be trained once and can be evaluated
8
Details of our translation system are provided in the appendix.
9
Note that even human translated test sets may underestimate a
model’s true cross-lingual generalization ability as such transla-
tionese has been shown to be less lexically diverse than naturally
composed language (Koppel & Ordan, 2011).