机器学习中的优化算法解析

4星 · 超过85%的资源需积分: 5 37 浏览量更新于2024-07-14 收藏 6.24MB PDF 举报

"First-Order and Stochastic Optimization Methods for Machine Learning.pdf" 本书是Springer Series in the Data Sciences系列的一部分，由Guanghui Lan撰写，专注于机器学习中的第一阶和随机优化方法。这本书旨在为学生和研究人员提供一个清晰的视角，以理解并应用这些优化技术在解决实际问题中的作用。该系列的编委会由多所知名大学和研究机构的专家组成，确保了内容的专业性和权威性。优化是机器学习的核心部分，因为它涉及到找到最佳模型参数以最大化预测性能或最小化损失函数。第一阶优化方法主要依赖于梯度信息，如梯度下降法，它是许多机器学习算法的基础，包括线性回归、逻辑回归和神经网络。这些方法通常易于理解和实现，但可能在高维空间中收敛速度较慢，尤其是在面对大规模数据集时。随机优化方法，如随机梯度下降（SGD），是在大数据场景下常用的一种策略。与传统的梯度下降不同，SGD每次迭代只考虑数据的一个小批量或单个样本，这大大减少了计算成本，使得在大量数据上训练大型模型成为可能。然而，随机性可能导致收敛速度不一致，可能需要额外的技巧，如动量法、自适应学习率调整（如Adagrad、RMSprop、Adam等）来改善其性能。书中可能涵盖了以下主题： 1. 梯度下降法：包括基本的梯度下降、批量梯度下降、梯度下降的变种，以及如何在实际应用中避免陷入局部最优。 2. 随机梯度下降：解释SGD的工作原理，如何减少计算复杂性，以及SGD在深度学习中的应用。 3. 动量法：介绍动量项如何加速收敛，如Nesterov加速梯度（NAG）。 4. 自适应学习率方法：如Adagrad、RMSprop、Adam等，它们根据参数的历史梯度调整学习率，以适应不同的参数更新速度。 5. 批量大小的选择：讨论批量大小对优化过程的影响，包括收敛速度和内存消耗。 6. 共轭梯度和拟牛顿法：对于无约束优化问题，这些二阶方法可以更快地收敛，但计算成本更高。 7. 鲁棒优化：处理噪声和异常值，以及在非凸优化问题中的应用。 8. 实战案例：通过实例展示优化算法在实际机器学习任务中的应用，可能包括图像识别、自然语言处理等领域的模型训练。 9. 实现与工具：可能介绍优化库（如TensorFlow、PyTorch）的使用，以及如何在实践中实施这些算法。通过深入阅读本书，读者将不仅理解优化的基本概念，还能掌握如何在复杂的机器学习项目中有效地应用这些方法，提高模型的训练效率和性能。

1.1 Linear Regression 3

Thus the minimizer of (1.1.1) is given by

∗

=(U

−1

The ordinary least square regression is among very few machine learning models

that has an explicit solution. Note, however, that to compute

∗

, one needs to com-

pute the inverse of an (n + 1) ×(n + 1) matrix (U

U). If the dimension of n is big,

computing the inverse of a large matrix can still be computationally expensive.

The formulation of the optimization problem in (1.1.1) follows a rather intuitive

approach. In the sequel, we provide some statistical reasoning about this formula-

tion. Let us denote

(i)

= v

(i)

−

(i)

,i = 1,...,N. (1.1.2)

In other words,

(i)

denotes the error associated with approximating v

(i)

Moreover, assume that

(i)

, i = 1,...,N, are i.i.d. (independently and identically dis-

tributed) according to a Gaussian (or Normal) distribution with mean 0 and variance

. Then, the density of

(i)

is then given by

(i)

√

πσ

exp



−

(

(i)

)



Using (1.1.2) in the above equation, we have

p(v

(i)

;

√

πσ

exp



−

(i)

−

(i)

)



. (1.1.3)

Here, p(v

(i)

;

) denotes the distribution of the output v

(i)

given input u

(i)

and

parameterized by

Given the input variables u

(i)

and output v

(i)

, i = 1,...,N, the likelihood function

with respect to (w.r.t.) the parameters

is deﬁned as

) :=

∏

i=1

p(v

(i)

;

∏

i=1

√

πσ

exp



−

(i)

−

(i)

)



The principle of

maximum likelihood

tells us that we should choose

to maximize

the likelihood L(

), or equivalently, the

log likelihood

) := log L(

)

∑

i=1

log



√

πσ

exp



−

(i)

−

(i)

)



= N log

√

πσ

−

∑

i=1

(i)

−

(i)

)

This is exactly the ordinary least square regression problem, i.e., to minimize

∑

i=1

(i)

−

(i)

)

w.r.t.

. The above reasoning tells us that under certain proba-

bilistic assumptions, the ordinary least square regression is the same as maximum

likelihood estimation. It should be noted, however, that the probabilistic assump-

tions are by no means necessary for least-squares to be a rational procedure for

regression.

4 1 Machine Learning Models

1.2 Logistic Regression

Let us come back to the previous example. Suppose that Julie only cares about

whether she will like the restaurant “Bamboo Garden” or not, rather her own rat-

ings. Moreover, she only recorded some historical data indicating whether she likes

or dislikes some restaurants, as shown in Table 1.2. These records are also visualized

in Fig. 1.1, where each restaurant is represented by a green “O” or a red “X,” corre-

sponding to whether Julie liked or disliked the restaurant, respectively. The question

is: with the rating of 3 from both of her friends, will Julie like Bamboo Garden? Can

she use the past data to come up with a reasonable decision?

Table 1.2 Historical ratings for the restaurants

Restaurant Judy’s rating Jim’s rating Julie likes?

Goodfellas 1 5 No

Hakkasan 4.5 4 Yes

··· ··· ··· ···

Bamboo Garden 3 3 ?

Similar to the regression model, the input values are still denoted by U =

(1)

;...; u

(N)

), i.e., the ratings given by Judy and Jim. But the output values

are now binary, i.e., v

(i)

∈{0,1}, i = 1,...,N.Herev

(i)

= 1 means that Julie likes

the i-th restaurant and v

(i)

= 0 means that she dislikes the restaurant. Julie’s goal is

to come up with a decision function h(u) to approximate these binary variables v.

This type of machine learning task is called

binary classiﬁcation

Julie’s decision function can be as simple as a weighted linear combination of

her friends’ ratings:

(u)=

+ ...+

(1.2.1)

with n = 2. One obvious problem with the decision function in (1.2.1) is that its

values can be arbitrarily large or small. On the other hand, Julie wishes its values to

fall between 0 and 1 because those represent the range of v. A simple way to force

h to fall within 0 and 1 is to map the linear decision function

u through another

function called the sigmoid (or logistic) function

g(z)=

1+exp(−z)

(1.2.2)

and deﬁne the decision function as

(u)=g(

u)=

1+exp(−

. (1.2.3)

Note that the range of the sigmoid function is given by (0,1),asshownin

Fig. 1.2.

剩余590页未读，继续阅读

Dreama_CS

粉丝: 4
资源: 11

机器学习中的优化算法解析

Convex Optimization for Machine Learning.pdf

Optimization Methods for Large-Scale Machine Learning

MIT Optimization for Machine Learning.

能推荐几篇遗传算法优化的文献吗

动手学深度学习参考文献的格式

计算机毕设英文参考文献

Derive the stochastic gradient descent algorithm

rank-based stochastic pooling

Robbins-Monro setting

stochastic control in continuous-time and application in finance

最新资源