XTREME: A Benchmark for Evaluating Cross-lingual generalization
using cosine similarity.
4
Tatoeba
We use the Tatoeba dataset (Artetxe & Schwenk,
2019), which consists of up to 1,000 English-aligned sen-
tence pairs covering 122 languages. We find the nearest
neighbour using cosine similarity and calculate error rate.
3.3. Languages
As noted in Section 3.1, we choose our target languages
based on availability of monolingual data, and typological
diversity. We use the number of articles in Wikipedia as a
proxy for the amount of monolingual data available online.
In order to strike a balance between language diversity and
availability of monolingual data, we select all languages
out of the top 100 Wikipedias
5
with the most articles as of
December 2019.
6
We first select all languages that appear
in at least three of our benchmark datasets. This leaves
us with 19 languages, most of which are Indo-European
or major world languages. We now select 21 additional
languages that appear in at least one dataset and come from
less represented language families. Wherever possible, we
choose at least two languages per family.
7
In total, XTREME covers the following 40 languages (shown
with their ISO 639-1 codes for brevity) belonging to 12
language families and two isolates: af, ar, bg, bn, de, el, en,
es, et, eu, fa, fi, fr, he, hi, hu, id, it, ja, jv, ka, kk, ko, ml, mr,
ms, my, nl, pt, ru, sw, ta, te, th, tl, tr, ur, vi, yo, and zh. We
provide a detailed overview of these languages in terms of
their number of Wikipedia articles, linguistic features, and
coverage in XTREME in the appendix.
While XTREME covers these languages in the sense that
there is gold standard data in at least one task in each lan-
guage, this does not mean that it covers all aspects of each
language that are necessary for transfer. Languages may re-
veal different characteristics based on the task, domain, and
register in which they are used. XTREME thus only serves
as a glimpse into a model’s true cross-lingual generalization
capability.
3.4. Pseudo test data for analyses
XTREME covers 40 languages overall. Evaluation across the
majority of languages is only possible for a subset of tasks,
i.e. POS, NER, and Tatoeba. As additional diagnositics and
to enable a broader comparison across languages for a more
4
Results can be improved using more sophisticated similarity
metrics (Artetxe & Schwenk, 2019).
5
https://meta.wikimedia.org/wiki/List_of_
Wikipedias
6
This also has the benefit that they are covered by state-of-the-
art methods such as mBERT and XLM.
7
For the Austro-Asiatic, Kartvelian, and Kra-Dai families as
well as for isolates, we only obtain one language.
diverse set of tasks, we automatically translate the English
portions of a representative classification and QA task to the
remaining languages using an in-house translation system.
8
We choose XNLI and XQuAD as both have test sets that are
translations of the English data by professional translators.
We first verify that performance on the translated test sets
is a good proxy for performance on the gold standard test
sets. We report the detailed results in the appendix. For
XQuAD, the automatically translated test sets underesti-
mate mBERT’s true performance by 3.0 F1 / 0.2 EM points,
similar to the 2.6 F1 points reported by Agi
´
c & Schluter
(2018) when translating the test data to other languages.
9
For XNLI, the automatically translated test sets overestimate
the true prediction accuracy by 2.4 points. In order to mea-
sure the translation quality between the human-translated
test data and our pseudo test data, we compute the BLEU
score, and the chrF score (Popovi
´
c, 2015), which is suitable
for measuring the translation quality of some languages such
as Chinese and Russian. For the 14 languages in XNLI, we
obtain average scores of 34.2 BLEU and 58.9 chrF scores
on our pseudo test data compared to the reference transla-
tions, which correlate with a Pearson’s
ρ
of 0.57 and 0.28
respectively with mBERT performance.
Translating the English data to the remaining languages
yields 40-way parallel pseudo test data that we employ for
analyses in Section 5.
4. Experiments
4.1. Training and evaluation setup
XTREME focuses on the evaluation of multilingual represen-
tations. We do not place any restriction on the amount or
nature of the monolingual data used for pretraining multi-
lingual representations. However, we request authors to be
explicit about the data they use for training, in particular
any cross-lingual signal. In addition, we suggest authors
should not use any additional labelled data in the target task
beyond the one that is provided.
For evaluation, we focus on zero-shot cross-lingual trans-
fer with English as the source language as this is the most
common setting for the evaluation of multilingual represen-
tations and as many tasks only have training data available in
English. Although English is not generally the best source
language for cross-lingual transfer for all target languages
(Lin et al., 2019), this is still the most practically useful
setting. A single source language also facilitates evaluation
as models only need to be trained once and can be evaluated
8
Details of our translation system are provided in the appendix.
9
Note that even human translated test sets may underestimate a
model’s true cross-lingual generalization ability as such transla-
tionese has been shown to be less lexically diverse than naturally
composed language (Koppel & Ordan, 2011).
剩余19页未读，继续阅读
syp_net
- 粉丝: 158
- 资源: 1187
我的内容管理
展开
最新资源
- 多模态联合稀疏表示在视频目标跟踪中的应用
- Kubernetes资源管控与Gardener开源软件实践解析
- MPI集群监控与负载平衡策略
- 自动化PHP安全漏洞检测：静态代码分析与数据流方法
- 青苔数据CEO程永：技术生态与阿里云开放创新
- 制造业转型： HyperX引领企业上云策略
- 赵维五分享：航空工业电子采购上云实战与运维策略
- 单片机控制的LED点阵显示屏设计及其实现
- 驻云科技李俊涛：AI驱动的云上服务新趋势与挑战
- 6LoWPAN物联网边界路由器：设计与实现
- 猩便利工程师仲小玉：Terraform云资源管理最佳实践与团队协作
- 类差分度改进的互信息特征选择提升文本分类性能
- VERITAS与阿里云合作的混合云转型与数据保护方案
- 云制造中的生产线仿真模型设计与虚拟化研究
- 汪洋在PostgresChina2018分享：高可用 PostgreSQL 工具与架构设计
- 2018 PostgresChina大会：阿里云时空引擎Ganos在PostgreSQL中的创新应用与多模型存储
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
