Published as a conference paper at ICLR 2019
jointly by back-propagation (BP). This training procedure is different from previous meta-learning
algorithms such as network architecture search (Colson et al., 2007; Liu et al., 2018; Pham et al.,
2018). In previous work, Φ represents as a set of network modules with different learning capacities,
where Θ and Φ were optimized in two BP stages iteratively by using two training sets that are non-
overlapped. For example, previous work divided an entire training set into a training and a validation
set. However, if Θ and Φ in previous work are optimized in the same set of training data, Φ would
choose the module with large complexity to overfit these data. In contrast, SN essentially prevents
overfitting by choosing normalizers to improve both learning and generalization ability as discussed
below.
Analyses of SN. To understand SN, we theoretically compare SN with BN, IN, and LN by repre-
senting them using weight normalization (WN) (Salimans & Kingma, 2016) that is independent of
mean and variance. WN is computed as v
w
T
x
kwk
2
, where w and x represent a filter and an image patch.
WN normalizes the norm of each filter to 1 and rescales to v.
Remark 1. Let x be an image patch with zero mean and unit variance, w
i
be a filter of the i-
th channel, i ∈ {1, 2, ..., C}, and v be the filter norm of WN. Eqn.(1) can be rewritten by
ˆ
h
in
=
γ
w
i
T
x
kw
i
k
2
+β;
ˆ
h
bn
= γ
w
i
T
x
kw
i
k
2
+β, s.t. γ ≤ v; and
ˆ
h
ln
= γ
w
i
T
x
kw
i
k
2
+
P
C
j6=i
kw
j
k
2
+β. By combining them,
SN in Eqn.(3) can be reformulated by
ˆ
h
sn
= w
in
ˆ
h
in
+w
bn
ˆ
h
bn
+w
ln
ˆ
h
ln
= γ
w
i
T
x
kw
i
k
2
+w
ln
P
C
j6=i
kw
j
k
2
+
β, s.t. w
bn
γ ≤ v, where w
in
, w
ln
, w
bn
are the weights and γ ≤ v is a constraint.
w
1
w
2
γ
1
w
1
w
2
LN
SN
w
1
w
2
w
1
w
2
IN
v
w
1
w
2
BN
v
v
v
γ
1
γ
1
γ
1
γ
2
γ
2
γ
2
γ
2
Figure 3: Geometric view of directions and
lengths of the filters in IN, BN, LN, and SN
by comparing them to WN.
Remark 1 simplifies SN in Eqn.(3), enabling us to com-
pare different normalizers geometrically by formulating
them with respect to WN. In Fig.3,
ˆ
h
in
of IN can be
computed similarly to WN with an additional bias β,
where the norms of all filters are normalized to 1 and then
rescaled to γ. As γ and v have the same learning dynamic,
the length of γ would be identically to v (see w
1
, w
2
of
IN). Moreover,
ˆ
h
bn
in BN can be rewritten as WN with
regularization over γ, making it shorter than v. Compared
to IN and LN, Luo et al. (2019) shows that the regulariza-
tion of BN improves generalization and increases angle
between filters, preventing them from coadaptation (see
w
1
, w
2
of BN). Furthermore,
ˆ
h
ln
in LN normalizes each
filter among channels where the filter norm is less con-
strained than IN and BN. That is, LN allows γ > v to
increase learning ability. Finally,
ˆ
h
sn
in SN inherits the
benefits from all of them and enables balance between
learning and generalization ability. For example, when the batch size is small, the random noise
from the batch statistics of BN would be too strong. SN is able to maintain performance by decreas-
ing w
bn
and increasing w
ln
, such that the regularization from BN is reduced and the learning ability
is enhanced by LN. This phenomenon is supported by our experiment. More results are provided in
Appendix B.
Variants of SN. SN has many extensions. For instance, a pretrained network with SN can be fine-
tuned by applying the argmax function on its control parameters where each normalization layer
selects only one normalizer, leading to sparse SN. For (8, 32) as an example, SN with sparsity
achieves top-1 accuracy of 77.0% in ImageNet with ResNet50, which is comparable to 76.9% of SN
without sparsity. Moreover, when the channels are divided into groups, each group could select its
own normalizer to increase representation power of SN. Our preliminary results suggest that group
SN performs better than SN in some senses. For instance, group SN with only two groups boosts
the top-1 accuracy of ResNet50 to 77.2% in ImageNet. The above two variants will be presented as
future work due to the length of paper. This work focuses on SN where the importance weights are
tied between channels.
Inference. When applying SN in test, the statistics of IN and LN are computed independently
for each sample, while BN uses batch average after training without computing moving average
in each iteration. Here batch average is performed in two steps. First, we freeze the parameters
of the network and all the SN layers, and feed the network with a certain number of mini-batches
4
剩余18页未读，继续阅读
电动汽车控制与安全
- 粉丝: 263
- 资源: 4186
我的内容管理
展开
最新资源
- 多模态联合稀疏表示在视频目标跟踪中的应用
- Kubernetes资源管控与Gardener开源软件实践解析
- MPI集群监控与负载平衡策略
- 自动化PHP安全漏洞检测：静态代码分析与数据流方法
- 青苔数据CEO程永：技术生态与阿里云开放创新
- 制造业转型： HyperX引领企业上云策略
- 赵维五分享：航空工业电子采购上云实战与运维策略
- 单片机控制的LED点阵显示屏设计及其实现
- 驻云科技李俊涛：AI驱动的云上服务新趋势与挑战
- 6LoWPAN物联网边界路由器：设计与实现
- 猩便利工程师仲小玉：Terraform云资源管理最佳实践与团队协作
- 类差分度改进的互信息特征选择提升文本分类性能
- VERITAS与阿里云合作的混合云转型与数据保护方案
- 云制造中的生产线仿真模型设计与虚拟化研究
- 汪洋在PostgresChina2018分享：高可用 PostgreSQL 工具与架构设计
- 2018 PostgresChina大会：阿里云时空引擎Ganos在PostgreSQL中的创新应用与多模型存储
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
