深度残差学习：构建一百层神经网络

需积分: 9 105 浏览量更新于2024-09-11 收藏 703KB PDF 举报

"本文主要探讨了微软研究院在深度神经网络领域的突破，提出了一种名为‘残差学习框架’的方法，使得训练深度显著超过以往的神经网络成为可能。研究发现，通过将层重新表述为学习输入层的残差函数，而不是学习无参考的函数，可以更容易地优化深度网络，并能从增加的深度中获得精度提升。在ImageNet数据集上，他们评估了深度高达152层的残差网络，比VGG网络更深但复杂度更低，且取得了3.57%的错误率，赢得了2015年ILSVRC分类任务的第一名。此外，还在CIFAR-10数据集上测试了100层和1000层的网络。" 在深度学习领域，神经网络的深度是影响模型性能的关键因素。传统的观点认为，更深的网络能够学习到更复杂的特征表示，从而提高模型在视觉识别任务上的表现。然而，随着网络深度的增加，训练难度也随之增大，容易出现梯度消失或梯度爆炸等问题，导致模型无法有效地学习。微软研究院的这篇工作提出了一个创新的解决方案，即残差学习框架（Deep Residual Learning）。这个框架的核心思想是将每一层网络视为学习输入层的残差，而不是直接学习该层的非线性变换。这种设计允许网络更容易地学习到输入层的微小变化，而不是从零开始构建复杂的特征。通过学习输入的“残差”，网络的优化过程变得更加顺畅，深度的增加不再成为性能下降的障碍。在实验部分，研究人员在ImageNet大规模视觉识别挑战赛（ILSVRC）的数据集上验证了这一方法。他们构建了深度达到152层的残差网络，相比之前流行的VGG网络（通常为16层或19层），尽管深度增加了8倍，但模型的计算复杂度并没有显著增加。这样的深度网络在ImageNet测试集上实现了3.57%的错误率，显著优于当时的其他模型，并最终赢得了ILSVRC 2015分类任务的冠军。此外，为了进一步验证深度网络的有效性，研究者还在CIFAR-10数据集上进行了实验，甚至尝试了100层和1000层的网络。这些实验结果表明，即使在小型数据集上，残差网络也能有效处理极深的结构，且性能依然保持稳定。这篇研究揭示了深度神经网络训练的新视角，即通过学习输入的残差来促进网络的优化，为构建更深层次的网络提供了可行路径。这一方法不仅推动了深度学习的发展，也对后续的网络架构如ResNet、 DenseNet等产生了深远影响。

way networks have not demonstrated accuracy gains with

extremely increased depth (e.g., over 100 layers).

3. Deep Residual Learning

3.1. Residual Learning

Let us consider H(x) as an underlying mapping to be

ﬁt by a few stacked layers (not necessarily the entire net),

with x denoting the inputs to the ﬁrst of these layers. If one

hypothesizes that multiple nonlinear layers can asymptoti-

cally approximate complicated functions

, then it is equiv-

alent to hypothesize that they can asymptotically approxi-

mate the residual functions, i.e., H(x) − x (assuming that

the input and output are of the same dimensions). So

rather than expect stacked layers to approximate H(x), we

explicitly let these layers approximate a residual function

F(x) := H(x) − x. The original function thus becomes

F(x)+x. Although both forms should be able to asymptot-

ically approximate the desired functions (as hypothesized),

the ease of learning might be different.

This reformulation is motivated by the counterintuitive

phenomena about the degradation problem (Fig. 1, left). As

we discussed in the introduction, if the added layers can

be constructed as identity mappings, a deeper model should

have training error no greater than its shallower counter-

part. The degradation problem suggests that the solvers

might have difﬁculties in approximating identity mappings

by multiple nonlinear layers. With the residual learning re-

formulation, if identity mappings are optimal, the solvers

may simply drive the weights of the multiple nonlinear lay-

ers toward zero to approach identity mappings.

In real cases, it is unlikely that identity mappings are op-

timal, but our reformulation may help to precondition the

problem. If the optimal function is closer to an identity

mapping than to a zero mapping, it should be easier for the

solver to ﬁnd the perturbations with reference to an identity

mapping, than to learn the function as a new one. We show

by experiments (Fig. 7) that the learned residual functions in

general have small responses, suggesting that identity map-

pings provide reasonable preconditioning.

3.2. Identity Mapping by Shortcuts

We adopt residual learning to every few stacked layers.

A building block is shown in Fig. 2. Formally, in this paper

we consider a building block deﬁned as:

y = F(x, {W

}) + x. (1)

Here x and y are the input and output vectors of the lay-

ers considered. The function F(x, {W

}) represents the

residual mapping to be learned. For the example in Fig. 2

that has two layers, F = W

σ(W

x) in which σ denotes

This hypothesis, however, is still an open question. See [28].

ReLU [29] and the biases are omitted for simplifying no-

tations. The operation F + x is performed by a shortcut

connection and element-wise addition. We adopt the sec-

ond nonlinearity after the addition (i.e., σ(y), see Fig. 2).

The shortcut connections in Eqn.(1) introduce neither ex-

tra parameter nor computation complexity. This is not only

attractive in practice but also important in our comparisons

between plain and residual networks. We can fairly com-

pare plain/residual networks that simultaneously have the

same number of parameters, depth, width, and computa-

tional cost (except for the negligible element-wise addition).

The dimensions of x and F must be equal in Eqn.(1).

If this is not the case (e.g., when changing the input/output

channels), we can perform a linear projection W

by the

shortcut connections to match the dimensions:

y = F(x, {W

}) + W

x. (2)

We can also use a square matrix W

in Eqn.(1). But we will

show by experiments that the identity mapping is sufﬁcient

for addressing the degradation problem and is economical,

and thus W

is only used when matching dimensions.

The form of the residual function F is ﬂexible. Exper-

iments in this paper involve a function F that has two or

three layers (Fig. 5), while more layers are possible. But if

F has only a single layer, Eqn.(1) is similar to a linear layer:

y = W

x + x, for which we have not observed advantages.

We also note that although the above notations are about

fully-connected layers for simplicity, they are applicable to

convolutional layers. The function F(x, {W

}) can repre-

sent multiple convolutional layers. The element-wise addi-

tion is performed on two feature maps, channel by channel.

3.3. Network Architectures

We have tested various plain/residual nets, and have ob-

served consistent phenomena. To provide instances for dis-

cussion, we describe two models for ImageNet as follows.

Plain Network. Our plain baselines (Fig. 3, middle) are

mainly inspired by the philosophy of VGG nets [41] (Fig. 3,

left). The convolutional layers mostly have 3×3 ﬁlters and

follow two simple design rules: (i) for the same output

feature map size, the layers have the same number of ﬁl-

ters; and (ii) if the feature map size is halved, the num-

ber of ﬁlters is doubled so as to preserve the time com-

plexity per layer. We perform downsampling directly by

convolutional layers that have a stride of 2. The network

ends with a global average pooling layer and a 1000-way

fully-connected layer with softmax. The total number of

weighted layers is 34 in Fig. 3 (middle).

It is worth noticing that our model has fewer ﬁlters and

lower complexity than VGG nets [41] (Fig. 3, left). Our 34-

layer baseline has 3.6 billion FLOPs (multiply-adds), which

is only 18% of VGG-19 (19.6 billion FLOPs).

剩余11页未读，继续阅读

qq_28016093

粉丝: 1
资源: 1

深度残差学习：构建一百层神经网络

使用numpy实现简单的双层神经网络

基于双隐含层BP神经网络的预测,bp神经网络隐含层层数的确定,matlab

深度残差学习：构建超过100层的神经网络

多层神经网络工作原理

构建多层神经网络：从零开始

深度学习神经网络和多层神经网络的区别

MLPClassifier构建单层神经网络

keras构建多层神经网络

通过二层神经系统 构建三层神经网络的代码

matlab全连接层神经网络

最新资源

通过二层神经系统构建三层神经网络的代码