WideResidualNetworks：深度学习中的宽度与性能优化

需积分: 9 92 浏览量更新于2024-08-29 收藏 383KB PDF 举报

"Resnet--1605.07146v1.pdf：这篇论文主要探讨了Wide Residual Networks（WRNs），是深度学习领域关于Residual Networks（ResNets）的一种改进版本。" 深度学习是现代人工智能的核心技术之一，尤其是在计算机视觉任务中，而Residual Networks（残差网络）在这一领域扮演了重要角色。ResNets通过引入残差块（Residual Blocks）解决了深层神经网络训练时的梯度消失和模型退化问题，使得网络能够训练到上千层仍然保持性能提升。然而，尽管ResNets能够处理更深的层次，但每次提高百分之一的准确率往往需要几乎翻倍的层数，这导致特征复用逐渐减少，训练速度变慢。论文作者Sergey Zagoruyko和Nikos Komodakis对此进行了深入研究，他们指出这种现象导致的“特征复用问题”阻碍了非常深的ResNets的进一步发展。为了解决这个问题，他们在ResNet块的架构上进行了详尽的实验分析，并提出了一种新的网络架构——Wide Residual Networks（WRNs）。WRNs的特点是减小网络的深度，同时增加宽度，这样可以在不牺牲性能的情况下，提高特征复用，从而加快训练速度。通过实验，WRNs显示出了优于传统瘦而深的ResNets的优势。例如，一个只有16层的宽残差网络在准确率和效率方面就超越了之前所有的深残差网络。这一发现强调了网络结构优化的重要性，即在追求深度的同时，宽度的调整也对模型性能有着显著影响。WRNs的成功表明，深度并不是唯一决定模型性能的关键因素，合理的网络宽度设计同样关键。这篇论文对深度学习领域的研究者和实践者提供了有价值的洞见，即在构建高效深度学习模型时，应当综合考虑网络的深度和宽度，找到最佳的架构平衡点，以实现最优的性能和训练效率。这种思想在后续的网络设计中得到了广泛应用，例如在MobileNets和 EfficientNets等模型中，都借鉴了WRNs的设计理念，通过宽度、深度和分辨率的动态调整来优化模型性能。

SERGEY ZAGORUYKO AND NIKOS KOMODAKIS: WIDE RESIDUAL NETWORKS 3

improve performance by increasing depth of thin networks one needs to add thousands of

layers in this case.

Use of dropout in ResNet blocks. Dropout was ﬁrst introduced in [25] and then was

adopted by many successful architectures as [14, 24] etc. It was mostly applied on top layers

that had a large number of parameters to prevent feature coadaptation and overﬁtting. It was

then mainly substituted by batch normalization [13] which was introduced as a technique to

reduce internal covariate shift in neural network activations by normalizing them to have spe-

ciﬁc distribution. It also works as a regularizer and the authors experimentally showed that a

network with batch normalization achieves better accuracy than a network with dropout. In

our case, as widening of residual blocks results in an increase of the number of parameters,

we studied the effect of dropout to regularize training and prevent overﬁtting. Previously,

dropout in residual networks was studied in [11] with dropout being inserted in the identity

part of the block, and the authors showed negative effects of that. Instead, we argue here

that dropout should be inserted between convolutional layers. Experimental results on wide

residual networks show that this leads to consistent gains, yielding even new state-of-the-

art results (e.g., 16-layer-deep wide residual network with dropout achieves 1.64% error on

SVHN).

In summary, the contributions of this work are as follows:

• We present a detailed experimental study of residual network architectures that thor-

oughly examines several important aspects of ResNet block structure.

• We propose a novel widened architecture for ResNet blocks that allows for residual

networks with signiﬁcantly improved performance.

• We propose a new way of utilizing dropout within deep residual networks so as to

properly regularize them and prevent overﬁtting during training.

• Last, we show that our proposed ResNet architectures achieve state-of-the-art results

on several datasets dramatically improving accuracy and speed of residual networks.

2 Wide residual networks

Residual block with identity mapping can be represented by the following formula:

l+1

= x

+ F (x

, W

) (1)

where x

l+1

and x

are input and output of the l-th unit in the network, F is a residual func-

tion and W

are parameters of the block. Residual network consists of sequentially stacked

residual blocks.

In [11] residual networks consisted of two type of blocks:

• basic - with two consecutive 3 × 3 convolutions with batch normalization and ReLU

preceding convolution: conv3 × 3-conv3 × 3 Fig.1(a)

• bottleneck - with one 3 × 3 convolution surrounded by dimensionality reducing and

expanding 1 × 1 convolution layers: conv1 × 1-conv3 × 3-conv1 × 1 Fig.1(b)

Compared to the original architecture [9] in [11] the order of batch normalization, acti-

vation and convolution in residual block was changed from conv-BN-ReLU to BN-ReLU-

conv. As the latter was shown to train faster and achieve better results we don’t consider

剩余12页未读，继续阅读

twlklove

粉丝: 0
资源: 12

WideResidualNetworks：深度学习中的宽度与性能优化

残差网络ResNet不同权重文件的选择与应用

MATLAB实现ResNet-50深度学习模型识别1000类别

Matlab实现ResNet-18用于ImageNet对象识别

残差网络resnet的三个权重文件，resnet-v1-50.pt，resnet-v1-101.pt，resnet-v1-152

dlib-face-recognition-resnet-model-v1.dat.zip

ResNet-50-model.keras.h5

keras-resnet-0.2.0.tar.gz

resnet-tensorflow-0.0.1.tar.gz

tf-lambda-resnet-0.1.4.tar.gz

Python库 | keras-resnet-0.1.0.tar.gz

最新资源