Published as a conference paper at ICLR 2019
Mat Mul
Linear
Linear Linear
Scale
SoftMax
Mat Mul
Linear
Q K V
input
(a) Self-attention
LConv
Linear
Linear
input
GLU
(b) Lightweight convolution
input
LConv
Linear
Linear
Linear
dynamic weights
GLU
(c) Dynamic convolution
Figure 2: Illustration of self-attention, lightweight convolutions and dynamic convolutions.
self-attention, LightConv has a fixed context window and it determines the importance of context el-
ements with a set of weights that do not change over time steps. We will show that models equipped
with lightweight convolutions show better generalization compared to regular convolutions and that
they can be competitive to state-of-the-art self-attention models (§6). This is surprising because
the common belief is that content-based self-attention mechanisms are necessary to obtaining state-
of-the-art results in natural language processing applications. Furthermore, the low computational
profile of LightConv enables us to formulate efficient dynamic convolutions (§4).
LightConv computes the following for the i-th element in the sequence and output channel c:
LightConv(X, W
d
cH
d
e,:
, i, c) = DepthwiseConv(X, softmax(W
d
cH
d
e,:
), i, c)
Weight sharing. We tie the parameters of every subsequent number of
d
H
channels, which reduces
the number of parameters by a factor of
d
H
. As illustration, a regular convolution requires 7,340,032
(d
2
× k) weights for d = 1024 and k = 7, a depthwise separable convolution has 7,168 weights
(d × k), and with weight sharing, H = 16, we have only 112 (H × k) weights. We will see that
this vast reduction in the number of parameters is crucial to make dynamic convolutions possible on
current hardware. Wang & Ji (2018) ties the weights of all channels (H = 1).
Softmax-normalization. We normalize the weights W ∈ R
H×k
across the temporal dimension k
using a softmax operation:
softmax(W )
h,j
=
exp W
h,j
P
k
j
0
=1
exp W
h,j
0
Module. Figure 2b shows the architecture of the module where we integrate LightConv. We first
apply an input projection mapping from dimension d to 2d, followed by a gated linear unit (GLU;
Dauphin et al. 2017), and the actual lightweight convolution. The GLU uses half of the inputs as
gates by applying sigmoid units and then computes a pointwise product with the other inputs. We
also apply an output projection of size W
O
∈ R
d×d
to the output of LightConv.
Regularization. We found DropConnect to be a good regularizer for the LightConv module
(Wan et al., 2013). Specifically, we drop every entry of the normalized weights softmax(W ) with
probability p and divide it by 1 −p during training. This amounts to removing some of the temporal
information within a channel.
Implementation. Existing CUDA primitives for convolutions did not perform very well to imple-
ment LightConv and we found the following solution faster on short sequences: We copy and expand
the normalized weights W ∈ R
H×k
to a band matrix of size BH × n × n, where B is the batch
size. We then reshape and transpose the inputs to size BH × n ×
d
H
, and perform a batch matrix
multiplication to get the outputs. We expect a dedicated CUDA kernel to be much more efficient.
3
剩余13页未读，继续阅读
无能为力就要努力
- 粉丝: 17
- 资源: 332
我的内容管理
展开
最新资源
- Hadoop生态系统与MapReduce详解
- MDS系列三相整流桥模块技术规格与特性
- MFC编程：指针与句柄获取全面解析
- LM06：多模4G高速数据模块，支持GSM至TD-LTE
- 使用Gradle与Nexus构建私有仓库
- JAVA编程规范指南：命名规则与文件样式
- EMC VNX5500 存储系统日常维护指南
- 大数据驱动的互联网用户体验深度管理策略
- 改进型Booth算法：32位浮点阵列乘法器的高速设计与算法比较
- H3CNE网络认证重点知识整理
- Linux环境下MongoDB的详细安装教程
- 压缩文法的等价变换与多余规则删除
- BRMS入门指南：JBOSS安装与基础操作详解
- Win7环境下Android开发环境配置全攻略
- SHT10 C语言程序与LCD1602显示实例及精度校准
- 反垃圾邮件技术：现状与前景
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
