深度残差学习：构建超过100层的神经网络

需积分: 27 160 浏览量更新于2024-09-07 收藏 755KB PDF 举报

"这篇学术论文深入探讨了深度残差学习在图像识别中的应用，提出了一种框架，使得训练更深的神经网络变得更加容易。通过将网络层重新定义为学习输入层的残差函数，而不是无参照的函数，解决了深度网络训练的困难。论文展示了这种残差网络在优化过程中的优势，并在ImageNet数据集上使用了152层的深度，比VGG网络更深但复杂度更低。这些残差网络的集成模型在ImageNet测试集上取得了3.57%的错误率，赢得了2015年ILSVRC分类任务的第一名。此外，该研究还分析了在CIFAR-10数据集上使用100和1000层的情况，强调了深度表示对于许多视觉识别任务的重要性。由于深度表示的增强，他们在COCO对象检测数据集上获得了28%的相对改进，在ILSVRC & COCO 2015竞赛中赢得了多项任务的第一名。" 本文的核心知识点包括： 1. **深度残差学习框架**：为了解决深度神经网络训练的困难，研究者提出了残差学习的概念，将网络层转化为学习输入层的残差函数。这种设计允许网络更有效地学习和传播梯度，从而克服深度网络的梯度消失或爆炸问题。 2. **网络深度与优化**：残差网络在优化过程中表现出优越性，能够随着深度的增加而保持或提高性能。这打破了以往的观点，即网络越深，训练难度越大。 3. **深度与准确性**：实验表明，深度残差网络可以从显著增加的深度中获益，提高了在ImageNet和CIFAR-10数据集上的分类准确性，证明了深度在网络表示能力上的关键作用。 4. **性能比较**：与VGG网络相比，尽管残差网络的深度更深，但其复杂度较低，说明了残差结构的效率。152层的残差网络在ImageNet上达到了比VGG网络更好的性能。 5. **实际应用效果**：深度残差学习不仅在分类任务上表现优异，还在ILSVRC & COCO 2015竞赛的多个任务（如图像检测、定位、分割）中赢得了第一名，显示了其在实际应用中的强大能力。 6. **深度对视觉识别的重要性**：论文强调，深度表示对于视觉识别任务至关重要。深度网络可以学习到更抽象、层次更丰富的特征，从而在目标检测和识别等任务中取得显著的提升。这篇论文为深度学习领域提供了一种创新的架构，即残差网络，它不仅解决了深度网络的训练难题，而且显著提升了模型的性能，特别是在深度学习的图像识别任务中。这一突破性的成果对后续的深度学习研究和应用产生了深远影响。

way networks have not demonstrated accuracy gains with

extremely increased depth (e.g., over 100 layers).

3. Deep Residual Learning

3.1. Residual Learning

Let us consider H(x) as an underlying mapping to be

ﬁt by a few stacked layers (not necessarily the entire net),

with x denoting the inputs to the ﬁrst of these layers. If one

hypothesizes that multiple nonlinear layers can asymptoti-

cally approximate complicated functions

, then it is equiv-

alent to hypothesize that they can asymptotically approxi-

mate the residual functions, i.e., H(x) − x (assuming that

the input and output are of the same dimensions). So

rather than expect stacked layers to approximate H(x), we

explicitly let these layers approximate a residual function

F(x) := H(x) − x. The original function thus becomes

F(x)+x. Although both forms should be able to asymptot-

ically approximate the desired functions (as hypothesized),

the ease of learning might be different.

This reformulation is motivated by the counterintuitive

phenomena about the degradation problem (Fig. 1, left). As

we discussed in the introduction, if the added layers can

be constructed as identity mappings, a deeper model should

have training error no greater than its shallower counter-

part. The degradation problem suggests that the solvers

might have difﬁculties in approximating identity mappings

by multiple nonlinear layers. With the residual learning re-

formulation, if identity mappings are optimal, the solvers

may simply drive the weights of the multiple nonlinear lay-

ers toward zero to approach identity mappings.

In real cases, it is unlikely that identity mappings are op-

timal, but our reformulation may help to precondition the

problem. If the optimal function is closer to an identity

mapping than to a zero mapping, it should be easier for the

solver to ﬁnd the perturbations with reference to an identity

mapping, than to learn the function as a new one. We show

by experiments (Fig. 7) that the learned residual functions in

general have small responses, suggesting that identity map-

pings provide reasonable preconditioning.

3.2. Identity Mapping by Shortcuts

We adopt residual learning to every few stacked layers.

A building block is shown in Fig. 2. Formally, in this paper

we consider a building block deﬁned as:

y = F(x, {W

}) + x. (1)

Here x and y are the input and output vectors of the lay-

ers considered. The function F(x, {W

}) represents the

residual mapping to be learned. For the example in Fig. 2

that has two layers, F = W

σ(W

x) in which σ denotes

This hypothesis, however, is still an open question. See [28].

ReLU [29] and the biases are omitted for simplifying no-

tations. The operation F + x is performed by a shortcut

connection and element-wise addition. We adopt the sec-

ond nonlinearity after the addition (i.e., σ(y), see Fig. 2).

The shortcut connections in Eqn.(1) introduce neither ex-

tra parameter nor computation complexity. This is not only

attractive in practice but also important in our comparisons

between plain and residual networks. We can fairly com-

pare plain/residual networks that simultaneously have the

same number of parameters, depth, width, and computa-

tional cost (except for the negligible element-wise addition).

The dimensions of x and F must be equal in Eqn.(1).

If this is not the case (e.g., when changing the input/output

channels), we can perform a linear projection W

by the

shortcut connections to match the dimensions:

y = F(x, {W

}) + W

x. (2)

We can also use a square matrix W

in Eqn.(1). But we will

show by experiments that the identity mapping is sufﬁcient

for addressing the degradation problem and is economical,

and thus W

is only used when matching dimensions.

The form of the residual function F is ﬂexible. Exper-

iments in this paper involve a function F that has two or

three layers (Fig. 5), while more layers are possible. But if

F has only a single layer, Eqn.(1) is similar to a linear layer:

y = W

x + x, for which we have not observed advantages.

We also note that although the above notations are about

fully-connected layers for simplicity, they are applicable to

convolutional layers. The function F(x, {W

}) can repre-

sent multiple convolutional layers. The element-wise addi-

tion is performed on two feature maps, channel by channel.

3.3. Network Architectures

We have tested various plain/residual nets, and have ob-

served consistent phenomena. To provide instances for dis-

cussion, we describe two models for ImageNet as follows.

Plain Network. Our plain baselines (Fig. 3, middle) are

mainly inspired by the philosophy of VGG nets [41] (Fig. 3,

left). The convolutional layers mostly have 3×3 ﬁlters and

follow two simple design rules: (i) for the same output

feature map size, the layers have the same number of ﬁl-

ters; and (ii) if the feature map size is halved, the num-

ber of ﬁlters is doubled so as to preserve the time com-

plexity per layer. We perform downsampling directly by

convolutional layers that have a stride of 2. The network

ends with a global average pooling layer and a 1000-way

fully-connected layer with softmax. The total number of

weighted layers is 34 in Fig. 3 (middle).

It is worth noticing that our model has fewer ﬁlters and

lower complexity than VGG nets [41] (Fig. 3, left). Our 34-

layer baseline has 3.6 billion FLOPs (multiply-adds), which

is only 18% of VGG-19 (19.6 billion FLOPs).

剩余11页未读，继续阅读

xinghaoyan

粉丝: 11
资源: 79

深度残差学习：构建超过100层的神经网络

Deep Residual Learning for Image Recognition.pdf

Deep Residual Learning for Image Recognition

Deep Residual Learning for Image Recognition（七月在线DL翻译组2017.4）

inet--ResNet--Deep Residual Learning for Image Recognition.pdf

Deep Residual Learning for Image Recognition 机翻

paper reading:Deep Residual Learning for Image Recognition

论文精读-Deep Residual Learning for Image Recognition

resnet ppt refer to Deep Residual Learning for Image Recognition

Deep Residual Learning for Image Recognition浅读与实现

Deep Residual Learning for Image Recognition 深度残差网络论文笔记

最新资源