pre-trained language or machine translation model as auxiliary features while training a supervised
model on the target task. This involves a substantial amount of new parameters for each separate
target task, whereas we require minimal changes to our model architecture during transfer.
Auxiliary training objectives
Adding auxiliary unsupervised training objectives is an alternative
form of semi-supervised learning. Early work by Collobert and Weston [
10
] used a wide variety of
auxiliary NLP tasks such as POS tagging, chunking, named entity recognition, and language modeling
to improve semantic role labeling. More recently, Rei [
50
] added an auxiliary language modeling
objective to their target task objective and demonstrated performance gains on sequence labeling
tasks. Our experiments also use an auxiliary objective, but as we show, unsupervised pre-training
already learns several linguistic aspects relevant to target tasks.
3 Framework
Our training procedure consists of two stages. The first stage is learning a high-capacity language
model on a large corpus of text. This is followed by a fine-tuning stage, where we adapt the model to
a discriminative task with labeled data.
3.1 Unsupervised pre-training
Given an unsupervised corpus of tokens
U = {u
1
, . . . , u
n
}
, we use a standard language modeling
objective to maximize the following likelihood:
L
1
(U) =
X
i
log P (u
i
|u
i−k
, . . . , u
i−1
; Θ) (1)
where
k
is the size of the context window, and the conditional probability
P
is modeled using a neural
network with parameters Θ. These parameters are trained using stochastic gradient descent [51].
In our experiments, we use a multi-layer Transformer decoder [
34
] for the language model, which is
a variant of the transformer [
62
]. This model applies a multi-headed self-attention operation over the
input context tokens followed by position-wise feedforward layers to produce an output distribution
over target tokens:
h
0
= UW
e
+ W
p
h
l
= transformer_block(h
l−1
)∀i ∈ [1, n]
P (u) = softmax(h
n
W
T
e
)
(2)
where
U = (u
−k
, . . . , u
−1
)
is the context vector of tokens,
n
is the number of layers,
W
e
is the token
embedding matrix, and W
p
is the position embedding matrix.
3.2 Supervised fine-tuning
After training the model with the objective in Eq. 1, we adapt the parameters to the supervised target
task. We assume a labeled dataset
C
, where each instance consists of a sequence of input tokens,
x
1
, . . . , x
m
, along with a label
y
. The inputs are passed through our pre-trained model to obtain
the final transformer block’s activation
h
m
l
, which is then fed into an added linear output layer with
parameters W
y
to predict y:
P (y|x
1
, . . . , x
m
) = softmax(h
m
l
W
y
). (3)
This gives us the following objective to maximize:
L
2
(C) =
X
(x,y )
log P (y|x
1
, . . . , x
m
). (4)
We additionally found that including language modeling as an auxiliary objective to the fine-tuning
helped learning by (a) improving generalization of the supervised model, and (b) accelerating
convergence. This is in line with prior work [
50
,
43
], who also observed improved performance with
such an auxiliary objective. Specifically, we optimize the following objective (with weight λ):
L
3
(C) = L
2
(C) + λ ∗ L
1
(C) (5)
Overall, the only extra parameters we require during fine-tuning are
W
y
, and embeddings for delimiter
tokens (described below in Section 3.3).
3
剩余11页未读，继续阅读
方案互联
- 粉丝: 18
- 资源: 926
下载权益
电子书特权
VIP文章
课程特权
VIP享7折，此内容立减3.57元 开通VIP
我的内容管理
展开
最新资源
- OptiX传输试题与SDH基础知识
- C++Builder函数详解与应用
- Linux shell (bash) 文件与字符串比较运算符详解
- Adam Gawne-Cain解读英文版WKT格式与常见投影标准
- dos命令详解：基础操作与网络测试必备
- Windows 蓝屏代码解析与处理指南
- PSoC CY8C24533在电动自行车控制器设计中的应用
- PHP整合FCKeditor网页编辑器教程
- Java Swing计算器源码示例：初学者入门教程
- Eclipse平台上的可视化开发：使用VEP与SWT
- 软件工程CASE工具实践指南
- AIX LVM详解：网络存储架构与管理
- 递归算法解析：文件系统、XML与树图
- 使用Struts2与MySQL构建Web登录验证教程
- PHP5 CLI模式：用PHP编写Shell脚本教程
- MyBatis与Spring完美整合：1.0.0-RC3详解
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
