![](https://csdnimg.cn/release/download_crawler_static/86285140/bg3.jpg)
Published as a conference paper at ICLR 2019
Mat Mul
Linear
Linear Linear
Scale
SoftMax
Mat Mul
Linear
Q K V
input
(a) Self-attention
LConv
Linear
Linear
input
GLU
(b) Lightweight convolution
input
LConv
Linear
Linear
Linear
dynamic weights
GLU
(c) Dynamic convolution
Figure 2: Illustration of self-attention, lightweight convolutions and dynamic convolutions.
self-attention, LightConv has a fixed context window and it determines the importance of context el-
ements with a set of weights that do not change over time steps. We will show that models equipped
with lightweight convolutions show better generalization compared to regular convolutions and that
they can be competitive to state-of-the-art self-attention models (§6). This is surprising because
the common belief is that content-based self-attention mechanisms are necessary to obtaining state-
of-the-art results in natural language processing applications. Furthermore, the low computational
profile of LightConv enables us to formulate efficient dynamic convolutions (§4).
LightConv computes the following for the i-th element in the sequence and output channel c:
LightConv(X, W
d
cH
d
e,:
, i, c) = DepthwiseConv(X, softmax(W
d
cH
d
e,:
), i, c)
Weight sharing. We tie the parameters of every subsequent number of
d
H
channels, which reduces
the number of parameters by a factor of
d
H
. As illustration, a regular convolution requires 7,340,032
(d
2
× k) weights for d = 1024 and k = 7, a depthwise separable convolution has 7,168 weights
(d × k), and with weight sharing, H = 16, we have only 112 (H × k) weights. We will see that
this vast reduction in the number of parameters is crucial to make dynamic convolutions possible on
current hardware. Wang & Ji (2018) ties the weights of all channels (H = 1).
Softmax-normalization. We normalize the weights W ∈ R
H×k
across the temporal dimension k
using a softmax operation:
softmax(W )
h,j
=
exp W
h,j
P
k
j
0
=1
exp W
h,j
0
Module. Figure 2b shows the architecture of the module where we integrate LightConv. We first
apply an input projection mapping from dimension d to 2d, followed by a gated linear unit (GLU;
Dauphin et al. 2017), and the actual lightweight convolution. The GLU uses half of the inputs as
gates by applying sigmoid units and then computes a pointwise product with the other inputs. We
also apply an output projection of size W
O
∈ R
d×d
to the output of LightConv.
Regularization. We found DropConnect to be a good regularizer for the LightConv module
(Wan et al., 2013). Specifically, we drop every entry of the normalized weights softmax(W ) with
probability p and divide it by 1 −p during training. This amounts to removing some of the temporal
information within a channel.
Implementation. Existing CUDA primitives for convolutions did not perform very well to imple-
ment LightConv and we found the following solution faster on short sequences: We copy and expand
the normalized weights W ∈ R
H×k
to a band matrix of size BH × n × n, where B is the batch
size. We then reshape and transpose the inputs to size BH × n ×
d
H
, and perform a batch matrix
multiplication to get the outputs. We expect a dedicated CUDA kernel to be much more efficient.
3