"深度学习优化算法解析：从理论到应用"

需积分: 0 53 浏览量更新于2024-03-21 收藏 18.84MB PDF 举报

Deep learning algorithms are at the forefront of artificial intelligence and involve complex optimization processes. The applications of deep learning are vast, from performing inference in models like PCA to training neural networks. Optimization is a crucial component in deep learning, whether it is for writing proofs, designing algorithms, or training deep models. One of the most challenging optimization problems in deep learning is neural network training. This process often requires a significant amount of time and computational resources, with researchers sometimes investing days to months and utilizing hundreds of machines to solve a single instance. The complexity of training neural networks lies in the large number of parameters that need to be optimized, as well as the non-convex nature of the optimization problem. Despite the challenges, optimization is essential for improving the performance of deep learning models. By fine-tuning the parameters of neural networks and optimizing the training process, researchers can enhance the accuracy and efficiency of these models. Optimization techniques such as stochastic gradient descent, Adam, and RMSprop play a critical role in training deep models and achieving state-of-the-art performance in various tasks such as image recognition, natural language processing, and reinforcement learning. In conclusion, optimization is a fundamental aspect of deep learning, powering the training and inference processes of complex models. While neural network training can be time-consuming and computationally intensive, the advancements in optimization algorithms have paved the way for groundbreaking innovations in artificial intelligence. By continuously refining and improving optimization techniques, researchers can unlock the full potential of deep learning and push the boundaries of what is possible in the field of AI.

CHAPTER 8. OPTIMIZATION FOR TRAINING DEEP MODELS

J w;b( )

Figure 8.3: The objective function for highly nonlinear deep neural networks or for

recurrent neural networks often contains sharp nonlinearities in parameter space resulting

from the multiplication of several parameters. These nonlinearities give rise to very

high derivatives in some places. When the parameters get close to such a cliﬀ region, a

gradient descent update can catapult the parameters very far, possibly losing most of the

optimization work that had been done.Figure adapted with permission from Pascanu

et al. ( ).2013a

The cliﬀ can be dangerous whether we approach it from above or from below,

but fortunately its most serious consequences can be avoided using the gradient

clipping heuristic described in Sec. .The basic idea is to recall that the10.11.1

gradient does not specify the optimal step size, but only the optimal direction

within an inﬁnitesimal region. When the traditional gradient descent algorithm

proposes to make a very large step, the gradient clipping heuristic intervenes to

reduce the step size to be small enough that it is less likely to go outside the region

where the gradient indicates the direction of approximately steepest descent. Cliﬀ

structures are most common in the cost functions for recurrent neural networks,

because such models involve a multiplication of many factors, with one factor

for each time step. Long temporal sequences thus incur an extreme amount of

multiplication.

8.2.5 Long-Term Dependencies

Another diﬃculty that neural network optimization algorithms must overcome arises

when the computational graph becomes extremely deep.Feedforward networks

with many layers have such deep computational graphs. So do recurrent networks,

described in Chapter ,which construct very deep computational graphs by10

289

CHAPTER 8. OPTIMIZATION FOR TRAINING DEEP MODELS

repeatedly applying the same operation at each time step of a long temporal

sequence. Repeated application of the same parameters gives rise to especially

pronounced diﬃculties.

For example, suppose that a computational graph contains a path that consists

of repeatedly multiplying by a matrix

. After

steps, this is equivalent to mul-

tiplying by

. Suppose that

has an eigendecomposition

V diag

(

)

−1

In this simple case, it is straightforward to see that



V λ Vdiag( )

−1



= ( )V diag λ

−1

. (8.11)

Any eigenvalues

that are not near an absolute value of will either explode if1

they are greater than in magnitude or vanish if they are less than in magnitude.1 1

The vanishing and exploding gradient problem refers to the fact that gradients

through such a graph are also scaled according to

diag

(

)

. Vanishing gradients

make it diﬃcult to know which direction the parameters should move to improve

the cost function, while exploding gradients can make learning unstable. The cliﬀ

structures described earlier that motivate gradient clipping are an example of the

exploding gradient phenomenon.

The repeated multiplication by

at each time step described here is very

similar to the power method algorithm used to ﬁnd the largest eigenvalue of a matrix

and the corresponding eigenvector. From this point of view it is not surprising

that

will eventually discard all components of

that are orthogonal to the

principal eigenvector of .W

Recurrent networks use the same matrix

at each time step, but feedforward

networks do not, so even very deep feedforward networks can largely avoid the

vanishing and exploding gradient problem ( , ).Sussillo 2014

We defer a further discussion of the challenges of training recurrent networks

until Sec. , after recurrent networks have been described in more detail.10.7

8.2.6 Inexact Gradients

Most optimization algorithms are primarily motivated by the case where we have

exact knowledge of the gradient or Hessian matrix. In practice, we usually only

have a noisy or even biased estimate of these quantities. Nearly every deep learning

algorithm relies on sampling-based estimates at least insofar as using a minibatch

of training examples to compute the gradient.

In other cases, the objective function we want to minimize is actually intractable.

When the objective function is intractable, typically its gradient is intractable as

well. In such cases we can only approximate the gradient. These issues mostly arise

290

CHAPTER 8. OPTIMIZATION FOR TRAINING DEEP MODELS

with the more advanced models in Part . For example, contrastive divergenceIII

gives a technique for approximating the gradient of the intractable log-likelihood

of a Boltzmann machine.

Various neural network optimization algorithms are designed to account for

imperfections in the gradient estimate. One can also avoid the problem by choosing

a surrogate loss function that is easier to approximate than the true loss.

8.2.7 Poor Correspondence between Local and Global Structure

Many of the problems we have discussed so far correspond to properties of the

loss function at a single point—it can be diﬃcult to make a single step if

(

) is

poorly conditioned at the current point

, or if

lies on a cliﬀ, or if

is a saddle

point hiding the opportunity to make progress downhill from the gradient.

It is possible to overcome all of these problems at a single point and still

perform poorly if the direction that results in the most improvement locally does

not point toward distant regions of much lower cost.

Goodfellow 2015et al. ( ) argue that much of the runtime of training is due to

the length of the trajectory needed to arrive at the solution. Fig. shows that8.2

the learning trajectory spends most of its time tracing out a wide arc around a

mountain-shaped structure.

Much of research into the diﬃculties of optimization has focused on whether

training arrives at a global minimum, a local minimum, or a saddle point, but in

practice neural networks do not arrive at a critical point of any kind. Fig. 8.1

shows that neural networks often do not arrive at a region of small gradient. Indeed,

such critical points do not even necessarily exist. For example, the loss function

−log p

(

y | x

;

) can lack a global minimum point and instead asymptotically

approach some value as the model becomes more conﬁdent. For a classiﬁer with

discrete

and

(

y | x

) provided by a softmax, the negative log-likelihood can

become arbitrarily close to zero if the model is able to correctly classify every

example in the training set, but it is impossible to actually reach the value of

zero. Likewise, a model of real values

(

y | x

) =

(

;

(

)

, β

−1

) can have negative

log-likelihood that asymptotes to negative inﬁnity—if

(

) is able to correctly

predict the value of all training set

targets, the learning algorithm will increase

without bound. See Fig. for an example of a failure of local optimization to8.4

ﬁnd a good cost function value even in the absence of any local minima or saddle

points.

Future research will need to develop further understanding of the factors that

inﬂuence the length of the learning trajectory and better characterize the outcome

291

CHAPTER 8. OPTIMIZATION FOR TRAINING DEEP MODELS

J ( )θ

Figure 8.4: Optimization based on local downhill moves can fail if the local surface does

not point toward the global solution. Here we provide an example of how this can occur,

even if there are no saddle points and no local minima. This example cost function

contains only asymptotes toward low values, not minima. The main cause of diﬃculty in

this case is being initialized on the wrong side of the “mountain” and not being able to

traverse it.In higher dimensional space, learning algorithms can often circumnavigate

such mountains but the trajectory associated with doing so may be long and result in

excessive training time, as illustrated in Fig. .8.2

of the process.

Many existing research directions are aimed at ﬁnding good initial points for

problems that have diﬃcult global structure, rather than developing algorithms

that use non-local moves.

Gradient descent and essentially all learning algorithms that are eﬀective for

training neural networks are based on making small, local moves. The previous

sections have primarily focused on how the correct direction of these local moves

can be diﬃcult to compute. We may be able to compute some properties of the

objective function, such as its gradient, only approximately, with bias or variance

in our estimate of the correct direction. In these cases, local descent may or may

not deﬁne a reasonably short path to a valid solution, but we are not actually

able to follow the local descent path. The objective function may have issues

such as poor conditioning or discontinuous gradients, causing the region where

the gradient provides a good model of the objective function to be very small. In

these cases, local descent with steps of size



may deﬁne a reasonably short path

to the solution, but we are only able to compute the local descent direction with

steps of size

δ 

. In these cases, local descent may or may not deﬁne a path

to the solution, but the path contains many steps, so following the path incurs a

292

CHAPTER 8. OPTIMIZATION FOR TRAINING DEEP MODELS

high computational cost. Sometimes local information provides us no guide, when

the function has a wide ﬂat region, or if we manage to land exactly on a critical

point (usually this latter scenario only happens to methods that solve explicitly

for critical points, such as Newton’s method). In these cases, local descent does

not deﬁne a path to a solution at all. In other cases, local moves can be too greedy

and lead us along a path that moves downhill but away from any solution, as in

Fig. , or along an unnecessarily long trajectory to the solution, as in Fig. .8.4 8.2

Currently, we do not understand which of these problems are most relevant to

making neural network optimization diﬃcult, and this is an active area of research.

Regardless of which of these problems are most signiﬁcant, all of them might be

avoided if there exists a region of space connected reasonably directly to a solution

by a path that local descent can follow, and if we are able to initialize learning

within that well-behaved region.This last view suggests research into choosing

good initial points for traditional optimization algorithms to use.

8.2.8 Theoretical Limits of Optimization

Several theoretical results show that there are limits on the performance of any

optimization algorithm we might design for neural networks ( ,Blum and Rivest

1992 Judd 1989 Wolpert and MacReady 1997; , ; , ). Typically these results have

little bearing on the use of neural networks in practice.

Some theoretical results apply only to the case where the units of a neural

network outputdiscrete values. However,mostneural network units output

smoothly increasing values that make optimization via local search feasible. Some

theoretical results show that there exist problem classes that are intractable, but

it can be diﬃcult to tell whether a particular problem falls into that class. Other

results show that ﬁnding a solution for a network of a given size is intractable, but

in practice we can ﬁnd a solution easily by using a larger network for which many

more parameter settings correspond to an acceptable solution. Moreover, in the

context of neural network training, we usually do not care about ﬁnding the exact

minimum of a function, but only in reducing its value suﬃciently to obtain good

generalization error.Theoretical analysis of whether an optimization algorithm

can accomplish this goal is extremely diﬃcult. Developing more realistic bounds

on the performance of optimization algorithms therefore remains an important

goal for machine learning research.

293

剩余230页未读，继续阅读

天使的梦魇

粉丝: 39
资源: 321

"深度学习优化算法解析：从理论到应用"

1.deep+learning课本-三巨头(下)1

深度学习教程1

deep-learning学习笔记1

TensorFlow+1.x+Deep+Learning+Cookbook-Packt+Publishing(2017).epub

TensorFlow 1.x Deep Learning Cookbook-PacktPublishing(2017).pdf

234.Deep-Learning-with-PyTorch-Tutorials__dragen1860.tar.gz

12.Deep-Learning-Code-Implementation:深度学习论文研究和代码实现

Bishop C. M., Bishop H. - Deep Learning. Foundations and Concepts - 2024

Deep+Learning+with+PyTorch-Packt+Publishing

Deep+Learning+with+PyTorch-Packt+Publishing(2018).zip

最新资源