克服训练难题：Ilya Sutskever的RNN优化与应用

Ilya

Sutskever

需积分: 10 49 浏览量更新于2024-07-19 收藏 3.13MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

"这篇资源是Ilya Sutskever在多伦多大学的博士论文，全英文版，专注于训练循环神经网络（RNN）的方法。论文提出了改进RNN的新策略，以解决处理长期依赖时间序列建模的问题。通过结合受限玻尔兹曼机（RBM）、无赫斯特优化器（Hessian-free optimizer）以及随机参数，Sutskever有效地提升了RNN的训练能力。" 在Ilya Sutskever的博士论文中，他深入探讨了训练循环神经网络（RNN）所面临的挑战及其解决方案。RNN是一种强大的序列模型，由于其在训练过程中的困难，过去在机器学习应用中并不常见。Sutskever在这篇论文中介绍了克服这些困难的方法，并展示了RNN在解决复杂问题上的潜力。首先，他提出了一种新的概率序列模型，将受限玻尔兹曼机（Restricted Boltzmann Machines, RBMs）与RNNs相结合。这种新模型在保持训练难度较低的同时，比类似的模型具有更强的表达能力。RBM是一种能够捕获数据隐藏结构的生成模型，与RNN结合可以更好地理解和生成复杂的序列数据。接着，Sutskever介绍了一种Hessian-free（HF）优化器的新变体。HF优化器是一种基于梯度的优化方法，不需计算Hessian矩阵，从而减少了计算复杂性。他证明了这种新变体可以训练RNN处理具有极长范围时间依赖的任务，这是以前被认为是几乎不可能解决的问题。这表明了RNN在处理时间序列分析，如语言建模、音频识别或自然语言处理等领域的巨大潜力。此外，Sutskever还应用HF优化器进行字符级的语言建模。在语言建模中，模型需要预测一个句子中的下一个字符，考虑到语言的复杂性和上下文依赖，这是一个极具挑战性的任务。HF优化器的高效性能使得RNN能够处理这种长期依赖性，从而提高了预测的准确性和模型的泛化能力。 Ilya Sutskever的博士论文提供了关于训练RNN的重要见解和方法，尤其是在处理长期依赖问题上的突破，这对深度学习领域的发展产生了深远影响。他的工作不仅改进了RNN的训练技术，还展示了这些技术在解决实际问题中的应用，为后来的研究者提供了宝贵的理论基础和实践指导。

资源详情

资源推荐

CHAPTER 2. BACKGROUND 8

1: distribute the input v across the input nodes Rz

for i ∈ I

2: for i from 1 to N if i 6∈ I do

3: Rx

← concat

j∈A

4: Rz

← f

) · Rx

5: end for

6: output Rz

, which is equal to F

(θ) · v

The correctness of this algorithm can likewise be proven by structural induction over the graph, proving

that Rz

= ∂z

/∂θ · v for each i, starting from i ∈ I and reaching node z

. Unlike backward differen-

tiation, forward differentiation does not need to store the state variables z

, since they can be computed

together with Rz

and be discarded once used.

Thus automatic differentiation can compute the Jacobian-vector products of any function expressible

as a computational graph at the cost of roughly two function evaluations. The Theano compiler (Bergstra

et al., 2010) computes derivatives in precisely this manner automatically and efﬁciently.

2.4 Feedforward Neural Networks

The Feedforward Neural Network (FNN) is the most basic and widely used artiﬁcial neural network. It

consists of a number of layers of artiﬁcial neurons (termed units) that are arranged into a layered con-

ﬁguration (ﬁg. 2.2). Of particular interest are deep neural networks, which are believed to be capable of

representing the highly complex functions that achieve high performance on difﬁcult perceptual prob-

lems such as vision and speech (Bengio, 2009). FNNs have achieved success in a number of domains

(e.g., Salakhutdinov and Hinton, 2009; Glorot et al., 2011; Krizhevsky and Hinton, 2011; Krizhevsky,

2010), most notably in large vocabulary continuous speech recognition (Mohamed et al., 2012), where

they were directly responsible for considerable improvements over previous highly-tuned, state-of-the-

art systems.

Formally, a feedforward neural network with ` hidden layers is parameterized by ` + 1 weight

matrices (W

, . . . , W

) and ` + 1 vectors of biases (b

, . . . , b

`+1

). The concatenation of the weight

matrices and the biases forms the parameter vector θ that fully speciﬁes the function computed by

the network. Given an input x, the feedforward neural network computes an output z as follows:

1: set z

← x

2: for i from 1 to ` + 1 do

3: x

← W

i−1

+ b

4: z

← e(x

)

5: end for

6: Output z ← z

Here e(·) is some nonlinear function such as the element-wise sigmoid sigmoid(x

) = 1/(1+exp(−x

)).

When e(·) is a sigmoid function, it can be used to implement Boolean logic, which allows deep and

wide feedforward neural networks to implement arbitrary Boolean circuits and hence arbitrary compu-

tation subject to some time and space constraints. But while we cannot hope to always train feedforward

neural networks so deep from labelled cases alone (provably so, under certain cryptographic assump-

tions (Kearns and Vazirani, 1994)

), wide and slightly deep networks (3-5 layers) are often easy to train

in practice with SGD while being moderately expressive.

Intuitively, given a public key, it is easy to generate a large number of (encryption, message) pairs. If circuits were

learnable, we could learn a circuit that could map an encryption to its secret message with high accuracy (since such a circuit

exists).

CHAPTER 2. BACKGROUND 9

Figure 2.2: The feedforward neural network.

FNNs are trained by minimizing the training error w.r.t. the parameters using a gradient method,

such as SGD or momentum.

Despite their representational power, deep FNNs have been historically considered very hard to

train, and until recently have not enjoyed widespread use. They became the subject of intense attention

thanks to the work of Hinton and Salakhutdinov (2006) and Hinton et al. (2006), who introduced the idea

of greedy layerwise pre-training, and successfully applied deep FNNs to a number of challenging tasks.

Greedy layerwise pre-training has since branched into a family of methods (Hinton et al., 2006; Hinton

and Salakhutdinov, 2006; Bengio et al., 2007), all of which train the layers of a deep FNN in order,

one at time, using an auxiliary objective, and then “ﬁne-tune” the network with standard optimization

methods such as stochastic gradient descent. More recently, Martens (2010) has attracted considerable

attention by showing that a type of truncated-Newton method called Hessian-free optimization (HF) is

capable of training deep FNNs from certain random initializations without the use of pre-training, and

can achieve lower errors for the various auto-encoding tasks considered in Hinton and Salakhutdinov

(2006). But recent results described in Chapter 7 show that even very deep neural networks can be

trained using an aggressive momentum schedule from well-chosen random initializations.

It is possible to implement the FNN with the computational graph formalism and to use backward

automatic differentiation to obtain the gradient (which is done if the FNN is implemented in Theano

(Bergstra et al., 2010)), but it is also straightforward to program the gradient directly:

1: dz

← dL(z

; y)/dz

2: for i from ` + 1 downto 1 do

3: dx

← e

) · dz

4: dz

i−1

← W

i−1

5: db

← dx

6: dW

i−1

← dx

i−1

7: end for

8: Output [dW

, . . . , dW

, db

, . . . , db

`+1

]

2.5 Recurrent Neural Networks

We are now ready to deﬁne the Recurrent Neural Network (RNN), the central object of study of this

thesis. The standard RNN is a nonlinear dynamical system that maps sequences to sequences. It is pa-

rameterized with three weight matrices and three bias vectors [W

, W

, b

, h

] whose con-

CHAPTER 2. BACKGROUND 11

colloquially as “the butterﬂy-effect”. The implication is that in an RNN, the derivative of the loss func-

tion at one time can be exponentially large with respect to the hidden activations at a much earlier time.

Thus the loss function is very sensitive to small changes, so it becomes effectively discontinuous.

RNNs also suffer from the vanishing gradient problem, ﬁrst described by Hochreiter (1991) and

Bengio et al. (1994). Consider the term

∂L(z

; y

)

∂W

, which is easy to analyze by inspecting line 10 of

the BPTT algorithm:

∂L(z

; y

)

∂W

t=1

t−1

(2.12)

where

τ=t+1

)



)

∂L(z

; y

)

∂p



(2.13)

If all the eigenvalues of W

are considerably smaller than 1, then the contributions dz

t−1

to dW

will rapidly diminish because dz

tends to zero exponentially as T −t increases. The latter phenomenon

is known as the vanishing gradient problem, and it is guaranteed to occur in any RNN that can store one

bit of information indeﬁnitely while being robust to at least some level of noise (Bengio et al., 1994),

a condition that ought to be satisﬁed by most RNNs that perform interesting computation. A vanishing

is undesirable, because it turns BPTT into truncated-BPTT, which is likely incapable of training

RNNs to exploit long-term temporal structure (see sec. 2.8.6).

The vanishing and the exploding gradient problems make it difﬁcult to optimize RNNs on sequences

with long-range temporal dependencies, and are possible causes for the abandonment of RNNs by ma-

chine learning researchers.

2.5.2 Recurrent Neural Networks as Generative models

Generative models are parameterized families of probability distributions that extrapolate a ﬁnite train-

ing set to a distribution over the entire space. They have many uses: good generative models of spec-

trograms can be used to synthesize speech; generative models of natural language can improve speech

recognition by deciding between words that the acoustic model cannot accurately distinguish; and im-

prove machine translation by evaluating the plausibility of a large number of candidate translations in

order to select the best one.

An RNN deﬁnes a generative model over sequences if the loss function satisﬁes L(z

; y

) = −log p(y

; z

)

for some parameterized family of distributions p(·; z) and if y

= v

t+1

. This deﬁnes the following dis-

tribution over sequences v

P (v

t−1

) ≡ P (v

; z

) (2.14)

P (v

) =

t=1

P (v

t−1

) ≡

t=1

P (v

; z

t−1

) (2.15)

where z

is an additional parameter vector that speciﬁes the distribution over the ﬁrst timestep of the

sequence, v

. This equation deﬁnes a valid distribution because z

is a function of v

and is independent

of v

t+1

. Samples from P (v

) can be obtained by sampling the conditional distribution v

∼ P (v

t−1

)

sequentially, by iterating from t = 1 to T . The loss function is precisely equivalent to the average

negative log probability of the sequences in the training set:

Train

(θ) = E

v∼S

[−log P

(v)] (2.16)

CHAPTER 2. BACKGROUND 12

where in this equation, the dependence of P on the parameters θ is explicit.

The log probability is a good objective function to optimize if we wish to ﬁt a distribution to data.

Assuming the data distribution D is precisely equal to P

∗

for some θ

∗

and that the mapping θ → P

is one-to-one, it is meaningful to discuss the rate at which our parameter estimates converge to θ

∗

the size of the training set increases. In this setting, if the log probability is uniformly bounded across

θ, the maximum likelihood estimator (i.e., the parameter setting maximizing eq. 2.16) is known to have

the fastest possible rate of convergence among all possible ways of mapping data to parameters (in a

minimax sense over the parameters) when the size of the training set is sufﬁciently large (Wasserman,

2004), which justiﬁes its use. In practice, the true data distribution D cannot usually be represented by

any setting of the parameters in the model P

∗

, but the average log probability objective is still used.

2.6 Overﬁtting

The term overﬁtting refers to the gap between the training error Train

(f) and the test error Test

(f).

We mentioned earlier that by restricting the functions of consideration f to a class of functions F we

control overﬁtting and address the generalization problem. Here we explain how limiting F accom-

plishes this.

Theorem 2.6.1. If F is ﬁnite and the loss is bounded L(z; y) ∈ [0, 1], then overﬁtting is uniformly

bounded with high probability over draws S from the training set (Kearns and Vazirani, 1994; Valiant,

1984):

S∼D

|S|

Test

(f) − Train

(f) ≤

log |F| + log 1/δ

|S|

for all f ∈ F

> 1 − δ (2.17)

The result suggests that when the training set is larger than log |F|, then every training error will be

close to every test error. The log |F| term in the bound formally justiﬁes the intuition that each training

case carries a constant number of bits about the best function in F.

Proof. We begin with the proof’s intuition. The central limit theorem ensures that the training error

Train

(f) is centred at Test

(f) and has Gaussian tails of size 1/

|S| (here we rely on L(z; y) ∈

[0, 1]). This means that, for a single function, the probability that its training error deviates from its test

error by more than log |F|/

|S| is exponentially small in log |F|. Hence the training error is fairly

unlikely to deviate by more than log |F|/

|S| for |F| functions simultaneously.

This intuition can be quantiﬁed by means of a one-sided Chernoff or Hoffeding bound for a single

ﬁxed f (Lugosi, 2004; Kearns and Vazirani, 1994; Valiant, 1984):

S∼D

|S|

[Test

(f) − Train

(f) > t] ≤ exp(−|S|t

) (2.18)

Applying the union bound, we can immediately control the maximal possible overﬁtting:

Pr [Test

(f) − Train

(f) > t for some f ∈ F] = Pr [∪

f∈F

{Test

(f) − Train

(f) > t}]

≤

f∈F

Pr [Test

(f) − Train

(f) > t]

≤ |F|exp(−|S|t

)

剩余100页未读，继续阅读

boyuancong

粉丝: 0
资源: 3

克服训练难题：Ilya Sutskever的RNN优化与应用

AlexNet笔记

Learning Recurrent Neural Networks with Hessian-Free Optimization

有没有好的参考文献提供一下

tell me more exact informations about OPENAI

Cifar10数据集

alexnet网络结构详解

alexnet是什么

图像识别有关的文献推荐

se注意力机制那一年提出的

列举20位为AI发展取到重要作用的科学家

alexnet模型详解

图像分类的特征提取可以参考哪篇文献？

chitGPT和chatGPT区别和创始人和创始团队

openai创始人是谁

人工智能深度学习参考文献

alexnet介绍 详细一点

动手学深度学习参考文献的格式

卷积神经网络经典论文

AlexNet网络模型介绍

最新资源

alexnet介绍详细一点