![](https://csdnimg.cn/release/download_crawler_static/19881042/bg4.jpg)
Published as a conference paper at ICLR 2019
jointly by back-propagation (BP). This training procedure is different from previous meta-learning
algorithms such as network architecture search (Colson et al., 2007; Liu et al., 2018; Pham et al.,
2018). In previous work, Φ represents as a set of network modules with different learning capacities,
where Θ and Φ were optimized in two BP stages iteratively by using two training sets that are non-
overlapped. For example, previous work divided an entire training set into a training and a validation
set. However, if Θ and Φ in previous work are optimized in the same set of training data, Φ would
choose the module with large complexity to overfit these data. In contrast, SN essentially prevents
overfitting by choosing normalizers to improve both learning and generalization ability as discussed
below.
Analyses of SN. To understand SN, we theoretically compare SN with BN, IN, and LN by repre-
senting them using weight normalization (WN) (Salimans & Kingma, 2016) that is independent of
mean and variance. WN is computed as v
w
T
x
kwk
2
, where w and x represent a filter and an image patch.
WN normalizes the norm of each filter to 1 and rescales to v.
Remark 1. Let x be an image patch with zero mean and unit variance, w
i
be a filter of the i-
th channel, i ∈ {1, 2, ..., C}, and v be the filter norm of WN. Eqn.(1) can be rewritten by
ˆ
h
in
=
γ
w
i
T
x
kw
i
k
2
+β;
ˆ
h
bn
= γ
w
i
T
x
kw
i
k
2
+β, s.t. γ ≤ v; and
ˆ
h
ln
= γ
w
i
T
x
kw
i
k
2
+
P
C
j6=i
kw
j
k
2
+β. By combining them,
SN in Eqn.(3) can be reformulated by
ˆ
h
sn
= w
in
ˆ
h
in
+w
bn
ˆ
h
bn
+w
ln
ˆ
h
ln
= γ
w
i
T
x
kw
i
k
2
+w
ln
P
C
j6=i
kw
j
k
2
+
β, s.t. w
bn
γ ≤ v, where w
in
, w
ln
, w
bn
are the weights and γ ≤ v is a constraint.
w
1
w
2
γ
1
w
1
w
2
LN
SN
w
1
w
2
w
1
w
2
IN
v
w
1
w
2
BN
v
v
v
γ
1
γ
1
γ
1
γ
2
γ
2
γ
2
γ
2
Figure 3: Geometric view of directions and
lengths of the filters in IN, BN, LN, and SN
by comparing them to WN.
Remark 1 simplifies SN in Eqn.(3), enabling us to com-
pare different normalizers geometrically by formulating
them with respect to WN. In Fig.3,
ˆ
h
in
of IN can be
computed similarly to WN with an additional bias β,
where the norms of all filters are normalized to 1 and then
rescaled to γ. As γ and v have the same learning dynamic,
the length of γ would be identically to v (see w
1
, w
2
of
IN). Moreover,
ˆ
h
bn
in BN can be rewritten as WN with
regularization over γ, making it shorter than v. Compared
to IN and LN, Luo et al. (2019) shows that the regulariza-
tion of BN improves generalization and increases angle
between filters, preventing them from coadaptation (see
w
1
, w
2
of BN). Furthermore,
ˆ
h
ln
in LN normalizes each
filter among channels where the filter norm is less con-
strained than IN and BN. That is, LN allows γ > v to
increase learning ability. Finally,
ˆ
h
sn
in SN inherits the
benefits from all of them and enables balance between
learning and generalization ability. For example, when the batch size is small, the random noise
from the batch statistics of BN would be too strong. SN is able to maintain performance by decreas-
ing w
bn
and increasing w
ln
, such that the regularization from BN is reduced and the learning ability
is enhanced by LN. This phenomenon is supported by our experiment. More results are provided in
Appendix B.
Variants of SN. SN has many extensions. For instance, a pretrained network with SN can be fine-
tuned by applying the argmax function on its control parameters where each normalization layer
selects only one normalizer, leading to sparse SN. For (8, 32) as an example, SN with sparsity
achieves top-1 accuracy of 77.0% in ImageNet with ResNet50, which is comparable to 76.9% of SN
without sparsity. Moreover, when the channels are divided into groups, each group could select its
own normalizer to increase representation power of SN. Our preliminary results suggest that group
SN performs better than SN in some senses. For instance, group SN with only two groups boosts
the top-1 accuracy of ResNet50 to 77.2% in ImageNet. The above two variants will be presented as
future work due to the length of paper. This work focuses on SN where the importance weights are
tied between channels.
Inference. When applying SN in test, the statistics of IN and LN are computed independently
for each sample, while BN uses batch average after training without computing moving average
in each iteration. Here batch average is performed in two steps. First, we freeze the parameters
of the network and all the SN layers, and feed the network with a certain number of mini-batches
4