简单循环单元：实现与卷积神经网络相当的训练速度
需积分: 19 189 浏览量
更新于2024-07-16
收藏 980KB PDF 举报
"Training rnns as fast as cnns.pdf" 是一篇关于提高循环神经网络（RNN）训练效率的论文，作者包括Tao Lei、Yu Zhang、Sida I. Wang、Hui Dai和Yoav Artzi，分别来自ASAPP Inc., Google Brain, Princeton University, 和Cornell University。
在这篇论文中，作者们提出了一个名为“Simple Recurrent Unit”（SRU）的新颖结构，旨在解决传统RNN在并行计算上的局限性。RNN由于其时间序列的特性，往往难以有效地进行并行计算，这限制了它们在大规模数据处理时的效率。SRU的设计目标是在保持模型表达能力的同时，提升其并行化程度，以适应高性能计算的需求。
SRU的核心特点是它的简洁性和高效性。它被设计成一个轻量级的循环单元，能够提供丰富的动态行为，同时允许高度并行的实现。为了便于深度模型的训练，SRU还采用了精心设计的初始化策略。通过这些优化，SRU在多个自然语言处理（NLP）任务上展示了其有效性。
论文中的实验结果表明，SRU在分类和问答数据集上的速度比优化后的cuDNN LSTM快5到9倍，并且在性能上超过了LSTM和卷积神经网络模型。此外，将SRU整合到Transformer架构中，平均提高了0.7个BLEU分数，这对于机器翻译任务是一个显著的提升。这表明SRU不仅在训练速度上有优势，而且在模型性能上也有竞争力。
"Training rnns as fast as cnns"这篇论文提出了SRU这一创新的RNN变体，旨在克服传统RNN的并行化难题，通过其高效的并行处理能力和深度学习优化，为NLP领域的模型训练提供了新的解决方案。SRU的出现，不仅提升了训练速度，还在多个任务上超越了现有标准模型，展示了其在实际应用中的巨大潜力。
3 Simple Recurrent Unit
We present and explain the design of Simple Re-
current Unit (SRU) in this section. A single layer
of SRU involves the following computation:
f
t
= σ (W
f
x
t
+ v
f
c
t−1
+ b
f
) (1)
c
t
= f
t
c
t−1
+ (1 − f
t
) (Wx
t
) (2)
r
t
= σ (W
r
x
t
+ v
r
c
t−1
+ b
r
) (3)
h
t
= r
t
c
t
+ (1 − r
t
) x
t
(4)
where W, W
f
and W
r
are parameter matrices
and v
f
, v
r
, b
f
and b
v
are parameter vectors to
be learnt during training. The complete architec-
ture decomposes to two sub-components: a light
recurrence (Equation 1 and 2) and a highway net-
work (Equation 3 and 4).
The light recurrence component successively
reads the input vectors x
t
and computes the se-
quence of states c
t
capturing sequential informa-
tion. The computation resembles other recurrent
networks such as LSTM, GRU and RAN (Lee
et al., 2017). Specifically, a forget gate f
t
controls
the information flow (Equation 1) and the state
vector c
t
is determined by adaptively averaging
the previous state c
t−1
and the current observation
Wx
t
according to f
t
(Equation 2).
One key design decision that differs from previ-
ous gated recurrent architectures is the way c
t−1
is used in the sigmoid gate. Typically, c
t−1
is
multiplied with a parameter matrix to compute f
t
,
e.g., f
t
= σ(W
f
x
t
+ V
f
c
t−1
+ b
f
). However,
the inclusion of V
f
c
t−1
makes it difficult to par-
allelize the state computation: each dimension of
c
t
and f
t
depends on all entries of c
t−1
, and the
computation has to wait until c
t−1
is fully com-
puted. To facilitate parallelization, our light recur-
rence component uses a point-wise multiplication
v
f
c
t−1
instead. With this simplification, each
dimension of the state vectors becomes indepen-
dent and hence parallelizable.
The highway network component (Srivastava
et al., 2015) facilitates gradient-based training of
deep networks. It uses the reset gate r
t
(Equation
3) to adaptively combine the input x
t
and the state
c
t
produced from the light recurrence (Equation
4), where (1 − r
t
) x
t
is a skip connection that
allows the gradient to directly propagate to the pre-
vious layer. Such connections have been shown to
improve scalability (Wu et al., 2016a; Kim et al.,
2016; He et al., 2016; Zilly et al., 2017).
The combination of the two components makes
the overall architecture simple yet expressive, and
easy to scale due to enhanced parallelization and
gradient propagation.
3.1 Parallelized Implementation
Despite the parallelization friendly design of SRU,
a naive implementation which computes equations
(1)–(4) for each step t sequentially would not
achieve SRU’s full potential. We employ two op-
timizations to enhance parallelism. The optimiza-
tions are performed in the context of GPU / CUDA
programming, but the general idea can be applied
to other parallel programming models.
We re-organize the computation of equations
(1)–(4) into two major steps. First, given the input
sequence {x
1
···x
L
}, we batch the matrix multi-
plications across all time steps. This significantly
improves the computation intensity (e.g. GPU uti-
lization). The batched multiplication is:
U
>
=
W
W
f
W
r
[x
1
, x
2
, ··· , x
L
] ,
where L is the sequence length, U ∈ R
L×3d
is
the computed matrix and d is the hidden state size.
When the input is a mini-batch of B sequences, U
would be a tensor of size (L, B, 3d).
The second step computes the remaining point-
wise operations. Specifically, we compile all
point-wise operations into a single fused CUDA
kernel and parallelize the computation across each
dimension of the hidden state. Algorithm 1 shows
the pseudo code of the forward function. The com-
plexity of this step is O(L ·B ·d) per layer, where
L is the sequence length and B is the batch size. In
contrast, the complexity of LSTM is O(L ·B ·d
2
)
because of the hidden-to-hidden multiplications
(e.g. Vh
t−1
), and each dimension can not be in-
dependently parallelized. The fused kernel also
reduces overhead. Without it, operations such as
sigmoid activation would each invoke a separate
function call, adding kernel launching latency and
more data moving costs.
The implementation of a bidirectional SRU is
similar: the matrix multiplications of both direc-
tions are batched, and the fused kernel handles and
parallelizes both directions at the same time.
3.2 Initialization
Proper parameter initialization can reduce gradient
propagation difficulties and hence have a positive
剩余14页未读，继续阅读
2022-07-13 上传
2018-04-08 上传
2021-03-30 上传
2019-09-09 上传
2021-04-19 上传
2021-03-13 上传
2019-09-12 上传
chde2Wang
- 粉丝: 428
- 资源: 22
我的内容管理
展开
最新资源
- 磁性吸附笔筒设计创新，行业文档精选
- Java Swing实现的俄罗斯方块游戏代码分享
- 骨折生长的二维与三维模型比较分析
- 水彩花卉与羽毛无缝背景矢量素材
- 设计一种高效的袋料分离装置
- 探索4.20图包.zip的奥秘
- RabbitMQ 3.7.x延时消息交换插件安装与操作指南
- 解决NLTK下载停用词失败的问题
- 多系统平台的并行处理技术研究
- Jekyll项目实战：网页设计作业的入门练习
- discord.js v13按钮分页包实现教程与应用
- SpringBoot与Uniapp结合开发短视频APP实战教程
- Tensorflow学习笔记深度解析：人工智能实践指南
- 无服务器部署管理器：防止错误部署AWS帐户
- 医疗图标矢量素材合集：扁平风格16图标(PNG/EPS/PSD)
- 人工智能基础课程汇报PPT模板下载
