ResNeSt: Split-Attention Networks 5
3.1 Split-Attention Block
Our Split-Attention block is a computational unit, consisting feature-map group
and split attention operations. Figure 1 (Right) depicts an overview of a Split-
Attention Block.
Feature-map Group. As in ResNeXt blocks [61], the input feature-map can
be divided into several groups along the channel dimension, and the number of
feature-map groups is given by a cardinality hyperparameter K. We refer to the
resulting feature-map groups as cardinal groups. We introduce a new radix hy-
perparameter R that dictates the number of splits within a cardinal group. Then
the block input X are split into G = KR groups along the channel dimension
X = {X
1
, X
2
, ...X
G
} as shown in Figure 1. We apply a series of transformations
{F
1
, F
2
, ...F
G
} to each individual group, then the intermediate representation
of each group can be represented as U
i
= F
i
(X
i
), for i ∈ {1, 2, ...G}.
Input 1 Input r
…
+
Input 2
Global pooling
Dense c’ + BN + ReLU
Dense c
Dense c
r-Softmax
x
Dense c
x x
+
(h, w, c)
(h, w, c)
(c, )
(c’, )
(c, )
(c, )
(h, w, c)
Fig. 2: Split-Attention within a
cardinal group. For easy visu-
alization in the figure, we use
c = C/K in this figure.
Split Attention in Cardinal Groups. Fol-
lowing [30, 38], a combined representation for
each cardinal group can be obtained by fusing
via an element-wise summation across mul-
tiple splits. The representation for k-th car-
dinal group is
ˆ
U
k
=
P
Rk
i=R(k−1)+1
U
i
.
ˆ
U
k
∈
R
H×W×C/K
for k ∈ 1, 2, ...K. Global contex-
tual information with embedded channel-wise
statistics can be gathered with global aver-
age pooling across spatial dimensions s
k
∈
R
C/K
[29,38]. Here the c-th component is cal-
culated as:
s
k
c
=
1
H × W
H
X
i=1
W
X
j=1
ˆ
U
k
c
(i, j). (1)
A weighted fusion of the cardinal group
representation V
k
∈ R
H×W×C/K
is aggre-
gated using channel-wise soft attention, where
each feature-map channel is produced using a
weighted combination over splits. The c-th channel is calculated as:
V
k
c
=
R
X
i=1
a
k
i
(c)U
R(k−1)+r
, (2)
where a
k
i
(c) denotes a (soft) assignment weight given by:
a
k
i
(c) =
exp(G
c
i
(s
k
))
P
R
j=0
exp(G
c
j
(s
k
))
if R > 1,
1
1+exp(−G
c
i
(s
k
))
if R = 1,
(3)
and mapping G
c
i
determines the weight of each split for the c-th channel based
on the global context representation s
k
.
剩余21页未读，继续阅读
佑林杉
- 粉丝: 10
- 资源: 28
我的内容管理
展开
最新资源
- 多模态联合稀疏表示在视频目标跟踪中的应用
- Kubernetes资源管控与Gardener开源软件实践解析
- MPI集群监控与负载平衡策略
- 自动化PHP安全漏洞检测：静态代码分析与数据流方法
- 青苔数据CEO程永：技术生态与阿里云开放创新
- 制造业转型： HyperX引领企业上云策略
- 赵维五分享：航空工业电子采购上云实战与运维策略
- 单片机控制的LED点阵显示屏设计及其实现
- 驻云科技李俊涛：AI驱动的云上服务新趋势与挑战
- 6LoWPAN物联网边界路由器：设计与实现
- 猩便利工程师仲小玉：Terraform云资源管理最佳实践与团队协作
- 类差分度改进的互信息特征选择提升文本分类性能
- VERITAS与阿里云合作的混合云转型与数据保护方案
- 云制造中的生产线仿真模型设计与虚拟化研究
- 汪洋在PostgresChina2018分享：高可用 PostgreSQL 工具与架构设计
- 2018 PostgresChina大会：阿里云时空引擎Ganos在PostgreSQL中的创新应用与多模型存储
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
