![](https://csdnimg.cn/release/download_crawler_static/12502992/bg6.jpg)
6 QIU XP, et al. Pre-trained Models for Natural Language Processing: A Survey March (2020)
Table 1: Loss Functions of Pre-training Tasks
Task Loss Function Description
LM L
LM
= −
T
X
t=1
log p(x
t
|x
<t
) x
<t
= x
1
, x
2
, · · · , x
t−1
.
MLM L
MLM
= −
X
ˆx∈m(x)
log p
ˆx|x
\m(x)
m
(
x
) and
x
\m(x)
denote the masked words from
x
and the rest
words respectively.
Seq2Seq MLM L
S2SMLM
= −
j
X
t=i
log p
x
t
|x
\x
i: j
, x
i:t−1
x
i: j
denotes an masked n-gram span from i to j in x.
PLM L
PLM
= −
T
X
t=1
log p(z
t
|z
<t
) z = perm(x) is a permutation of x with random order.
DAE L
DAE
= −
T
X
t=1
log p(x
t
|
ˆ
x, x
<t
)
ˆ
x is randomly perturbed text from x.
DIM L
DIM
= s(
ˆ
x
i: j
, x
i: j
) − log
X
˜
x
i: j
∈N
s(
ˆ
x
i: j
,
˜
x
i: j
) x
i: j
denotes an n-gram span from
i
to
j
in
x
,
ˆ
x
i: j
denotes a
sentence masked at position
i
to
j
, and
˜
x
i: j
denotes a randomly-
sampled negative n-gram from corpus.
NSP/SOP L
NSP/SOP
= − log p(t|x, y) t = 1 if x and y are continuous segments from corpus.
RTD L
RTD
= −
T
X
t=1
log p(y
t
|
ˆ
x) y
t
= 1( ˆx
t
= x
t
),
ˆ
x is corrupted from x.
1
x = [x
1
, x
2
, · · · , x
T
] denotes a sequence.
An improved solution is bidirectional LM (BiLM), which con-
sists of two unidirectional LMs: a forward left-to-right LM
and a backward right-to-left LM. For BiLM, Baevski et al.
[6]
proposed a two-tower model that the forward tower oper-
ates the left-to-right LM and the backward tower operates the
right-to-left LM.
3.1.2 Masked Language Modeling (MLM)
Masked language modeling (MLM) is first proposed by Taylor
[178]
in the literature, who referred to this as a Cloze task.
Devlin et al.
[36]
adapted this task as a novel pre-training task
to overcome the drawback of the standard unidirectional LM.
Loosely speaking, MLM first masks out some tokens from the
input sentences and then trains the model to predict the masked
tokens by the rest of the tokens. However, this pre-training
method will create a mismatch between the pre-training phase
and the fine-tuning phase because the mask token does not
appear during the fine-tuning phase. Empirically, to deal with
this issue, Devlin et al.
[36]
used a special
[MASK]
token 80%
of the time, a random token 10% of the time and the original
token 10% of the time to perform masking.
Sequence-to-Sequence MLM (Seq2Seq MLM)
MLM is
usually solved as classification problem. We feed the masked
sequences to a neural encoder whose output vectors are fur-
ther fed into a softmax classifier to predict the masked token.
Alternatively, we can use encoder-decoder (aka. sequence-to-
sequence) architecture for MLM, in which the encoder is fed
a masked sequence, and the decoder sequentially produces
the masked tokens in auto-regression fashion. We refer to
this kind of MLM as sequence-to-sequence MLM (Seq2Seq
MLM), which is used in MASS [
160
] and T5 [
144
]. Seq2Seq
MLM can benefit the Seq2Seq-style downstream tasks, such
as question answering, summarization, and machine transla-
tion.
Enhanced Masked Language Modeling (E-MLM)
Con-
currently, there are multiple research proposing different en-
hanced versions of MLM to further improve on BERT. Instead
of static masking, RoBERTa [
117
] improves BERT by dy-
namic masking.
UniLM [
39
,
8
] extends the task of mask prediction on three
types of language modeling tasks: unidirectional, bidirec-
tional, and sequence-to-sequence prediction. XLM [
27
] per-
forms MLM on a concatenation of parallel bilingual sentence
pairs, called Translation Language Modeling (TLM). Span-
BERT [
77
] replaces MLM with Random Contiguous Words
Masking and Span Boundary Objective (SBO) to integrate
structure information into pre-training, which requires the
system to predict masked spans based on span boundaries. Be-
sides, StructBERT [
193
] introduces the Span Order Recovery
task to further incorporate language structures.
Another way to enrich MLM is to incorporate external
knowledge (see Section 4.1).
3.1.3 Permuted Language Modeling (PLM)
Despite the wide use of the MLM task in pre-training, Yang
et al.
[209]
claimed that some special tokens used in the pre-
training of MLM, like
[MASK]
, are absent when the model is
applied on downstream tasks, leading to a gap between pre-
training and fine-tuning. To overcome this issue, Permuted
Language Modeling (PLM) [
209
] is a pre-training objective