SERGEY ZAGORUYKO AND NIKOS KOMODAKIS: WIDE RESIDUAL NETWORKS 3
thus seem to indicate that the main power of deep residual networks is in residual blocks, and
that the effect of depth is supplementary. We note that one can train even better wide resid-
ual networks that have twice as many parameters (and more), which suggests that to further
improve performance by increasing depth of thin networks one needs to add thousands of
layers in this case.
Use of dropout in ResNet blocks. Dropout was first introduced in [27] and then was
adopted by many successful architectures as [16, 26] etc. It was mostly applied on top layers
that had a large number of parameters to prevent feature coadaptation and overfitting. It was
then mainly substituted by batch normalization [15] which was introduced as a technique to
reduce internal covariate shift in neural network activations by normalizing them to have spe-
cific distribution. It also works as a regularizer and the authors experimentally showed that a
network with batch normalization achieves better accuracy than a network with dropout. In
our case, as widening of residual blocks results in an increase of the number of parameters,
we studied the effect of dropout to regularize training and prevent overfitting. Previously,
dropout in residual networks was studied in [13] with dropout being inserted in the identity
part of the block, and the authors showed negative effects of that. Instead, we argue here
that dropout should be inserted between convolutional layers. Experimental results on wide
residual networks show that this leads to consistent gains, yielding even new state-of-the-
art results (e.g., 16-layer-deep wide residual network with dropout achieves 1.64% error on
SVHN).
In summary, the contributions of this work are as follows:
• We present a detailed experimental study of residual network architectures that thor-
oughly examines several important aspects of ResNet block structure.
• We propose a novel widened architecture for ResNet blocks that allows for residual
networks with significantly improved performance.
• We propose a new way of utilizing dropout within deep residual networks so as to
properly regularize them and prevent overfitting during training.
• Last, we show that our proposed ResNet architectures achieve state-of-the-art results
on several datasets dramatically improving accuracy and speed of residual networks.
2 Wide residual networks
Residual block with identity mapping can be represented by the following formula:
x
l+1
= x
l
+ F (x
l
, W
l
) (1)
where x
l+1
and x
l
are input and output of the l-th unit in the network, F is a residual func-
tion and W
l
are parameters of the block. Residual network consists of sequentially stacked
residual blocks.
In [13] residual networks consisted of two type of blocks:
• basic - with two consecutive 3 × 3 convolutions with batch normalization and ReLU
preceding convolution: conv3 × 3-conv3 × 3 Fig.1(a)
• bottleneck - with one 3 × 3 convolution surrounded by dimensionality reducing and
expanding 1 × 1 convolution layers: conv1 × 1-conv3 × 3-conv1 × 1 Fig.1(b)
剩余14页未读，继续阅读
hywcxq
- 粉丝: 0
- 资源: 33
我的内容管理
展开
最新资源
- Lombok 快速入门与注解详解
- SpringSecurity实战：声明式安全控制框架解析
- XML基础教程：从数据传输到存储解析
- Matlab实现图像空间平移与镜像变换示例
- Python流程控制与运算符详解
- Python基础：类型转换与循环语句
- 辰科CD-6024-4控制器说明书：LED亮度调节与触发功能解析
- AE particular插件全面解析：英汉对照与关键参数
- Shell脚本实践：创建tar包、字符串累加与简易运算器
- TMS320F28335：浮点处理器与ADC详解
- 互联网基础与结构解析：从ARPANET到多层次ISP
- Redhat系统中构建与Windows共享的Samba服务器实战
- microPython编程指南：从入门到实践
- 数据结构实验：顺序构建并遍历链表
- NVIDIA TX2系统安装与恢复指南
- C语言实现贪吃蛇游戏基础代码
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
