自注意力与卷积层的关系探索
需积分: 30 126 浏览量
更新于2024-07-09
收藏 1.53MB PDF 举报
"这篇论文‘On the Relationship between Self-Attention and Convolutional Layers’在ICLR 2020会议上发表，探讨了自注意力机制与卷积层之间的关系。作者包括Jean-Baptiste Cordonnier, Andreas Loukas和Martin Jaggi，他们来自瑞士洛桑联邦理工学院（EPFL）。"
在计算机视觉领域，近期的研究趋势表明，注意力机制正逐渐被引入到模型中，挑战了传统卷积层作为核心构建块的地位。Ramachandran等人在2019年的研究表明，注意力机制可以完全替代卷积，并在视觉任务上达到最先进的性能。这引发了一个问题：学习到的注意力层是否运作方式与卷积层相似？
该论文提供了证据表明，注意力层确实可以执行类似卷积的操作，特别是当使用足够数量的注意力头时。具体来说，作者证明了一个多头自注意力层至少与任何卷积层一样具有表达能力。通过数值实验，他们展示了自注意力层对像素网格模式的关注方式与CNN层相似，这进一步证实了他们的分析。
在深度学习中，卷积层通常用于提取图像特征，通过在局部区域内进行滤波操作，捕捉空间上的局部相关性。而自注意力机制则允许模型在不同位置之间建立长距离依赖，从而捕获全局上下文信息。尽管它们在概念上有所不同，但这篇论文揭示了这两种方法在实际应用中可能有着更深层次的联系。
论文指出，即使注意力机制可以模拟卷积的效果，但这并不意味着它在所有情况下都能完全替代卷积。卷积层在处理图像数据时，由于其固有的平移等变性和参数共享特性，往往表现出色。而自注意力层虽然可以捕获更复杂的依赖关系，但在计算效率和内存需求方面可能会更高。
总结来说，这项研究为理解自注意力机制和卷积层之间的关系提供了新的视角，有助于我们更好地设计和优化计算机视觉模型。同时，这也提示我们在未来的研究中，可以探索如何结合这两种机制的优点，以创建更强大、更灵活的视觉模型。
Published as a conference paper at ICLR 2020
The theorem is proven constructively by selecting the parameters of the multi-head self-attention
layer so that the latter acts like a convolutional layer. In the proposed construction, the attention
scores of each self-attention head should attend to a different relative shift within the set ∆∆
K
=
{−bK/2c, . . . , bK/2c}
2
of all pixel shifts in a K × K kernel. The exact condition can be found in
the statement of Lemma 1.
Then, Lemma 2 shows that the aforementioned condition is satisfied for the relative positional en-
coding that we refer to as the quadratic encoding:
v
(h)
:= −α
(h)
(1, −2∆
(h)
1
, −2∆
(h)
2
) r
δ
:= (kδk
2
, δ
1
, δ
2
) W
qry
=W
key
:= 0
d
W
key
:= I (9)
The learned parameters ∆
(h)
= (∆
(h)
1
, ∆
(h)
2
) and α
(h)
determine the center and width of attention
of each head, respectively. On the other hand, δ = (δ
1
, δ
2
) is fixed and expresses the relative shift
between query and key pixels.
It is important to stress that the above encoding is not the only one for which the conditions of
Lemma 1 are satisfied. In fact, in our experiments, the relative encoding learned by the neural
network also matched the conditions of the lemma (despite being different from the quadratic en-
coding). Nevertheless, the encoding defined above is very efficient in terms of size, as only D
p
= 3
dimensions suffice to encode the relative position of pixels, while also reaching similar or better
empirical performance (than the learned one).
The theorem covers the general convolution operator as defined in eq. (17). However, machine
learning practitioners using differential programming frameworks (Paszke et al., 2017; Abadi et al.,
2015) might question if the theorem holds for all hyper-parameters of 2D convolutional layers:
• Padding: a multi-head self-attention layer uses by default the "SAME" padding while a
convolutional layer would decrease the image size by K − 1 pixels. The correct way to
alleviate these boundary effects is to pad the input image with bK/2c zeros on each side.
In this case, the cropped output of a MHSA and a convolutional layer are the same.
• Stride: a strided convolution can be seen as a convolution followed by a fixed pooling
operation—with computational optimizations. Theorem 1 is defined for stride 1, but a
fixed pooling layer could be appended to the Self-Attention layer to simulate any stride.
• Dilation: a multi-head self-attention layer can express any dilated convolution as each head
can attend a value at any pixel shift and form a (dilated) grid pattern.
Remark for the 1D case. Convolutional layers acting on sequences are commonly used in the lit-
erature for text (Kim, 2014), as well as audio (van den Oord et al., 2016) and time series (Franceschi
et al., 2019). Theorem 1 can be straightforwardly extended to show that multi-head self-attention
with N
h
heads can also simulate a 1D convolutional layer with a kernel of size K = N
h
with
min(D
h
, D
out
) output channels using a positional encoding of dimension D
p
≥ 2. Since we have
not tested empirically if the preceding construction matches the behavior of 1D self-attention in
practice, we cannot claim that it actually learns to convolve an input sequence—only that it has the
capacity to do so.
PROOF OF MAIN THEOREM
The proof follows directly from Lemmas 1 and 2 stated below:
Lemma 1. Consider a multi-head self-attention layer consisting of N
h
= K
2
heads, D
h
≥ D
out
and let f : [N
h
] → ∆∆
K
be a bijective mapping of heads onto shifts. Further, suppose that for
every head the following holds:
softmax(A
(h)
q,:
)
k
=
1 if f(h) = q − k
0 otherwise.
(10)
Then, for any convolutional layer with a K × K kernel and D
out
output channels, there exists
{W
(h)
val
}
h∈[N
h
]
such that MHSA(X) = Conv(X) for every X ∈ R
W ×H×D
in
.
4
剩余17页未读，继续阅读
2023-08-01 上传
2023-03-16 上传
2021-03-05 上传
2019-12-09 上传
2022-07-14 上传
2019-06-09 上传
2019-09-03 上传
2017-12-19 上传
2019-09-16 上传
潜夙
- 粉丝: 0
- 资源: 40
我的内容管理
展开
最新资源
- zlib-1.2.12压缩包解析与技术要点
- 微信小程序滑动选项卡源码模版发布
- Unity虚拟人物唇同步插件Oculus Lipsync介绍
- Nginx 1.18.0版本WinSW自动安装与管理指南
- Java Swing和JDBC实现的ATM系统源码解析
- 掌握Spark Streaming与Maven集成的分布式大数据处理
- 深入学习推荐系统：教程、案例与项目实践
- Web开发者必备的取色工具软件介绍
- C语言实现李春葆数据结构实验程序
- 超市管理系统开发：asp+SQL Server 2005实战
- Redis伪集群搭建教程与实践
- 掌握网络活动细节：Wireshark v3.6.3网络嗅探工具详解
- 全面掌握美赛：建模、分析与编程实现教程
- Java图书馆系统完整项目源码及SQL文件解析
- PCtoLCD2002软件：高效图片和字符取模转换
- Java开发的体育赛事在线购票系统源码分析
