高斯过程在机器学习中的应用探析

需积分: 10 36 浏览量更新于2024-07-26 1 收藏 675KB PDF 举报

"麻省理工2006年出版的《机械学习中的高斯过程》第三章，主要探讨了高斯过程在分类中的应用。该书由C.E.Rasmussen和C.K.I.Williams合著，是高斯过程机器学习领域的经典著作，可在www.GaussianProcess.org/gpml获取。第四章重点讲述了协方差函数，它是高斯过程预测器的核心部分，定义了数据点之间的相似性，对监督学习中的预测起到关键作用。" 高斯过程是机器学习中的一种概率模型，它提供了一种非参数方法来建模复杂的数据分布。在高斯过程中，每个输入点都对应一个随机变量，而整个输入集合上的随机变量集合则服从联合高斯分布。这使得我们可以对未知函数的输出进行概率预测。在描述高斯过程时，协方差函数（也称为核函数）起着至关重要的角色。协方差函数定义了输入空间中的距离度量，即两个输入点$x$和$x'$之间的相似度。根据高斯过程的性质，如果两个输入点$x$和$x'$在协方差函数下具有较大的相似度，那么它们对应的输出$y$和$y'$也更可能接近。因此，选择合适的协方差函数是高斯过程模型的关键。第4章“协方差函数”中，作者介绍了多种常用的协方差函数及其特性。这些基本术语和概念包括但不限于： 1. **线性协方差函数**：最简单的形式，通常用于表示输入之间的线性关系。 2. **平方指数协方差函数（高斯核）**：也称为RBF（Radial Basis Function）或高斯核，它提供了平滑的、全局的函数近似，广泛应用于各种问题。 3. **周期性协方差函数**：用于捕捉数据中的周期性模式，例如时间序列分析中的季节性。 4. ** Matérn 协方差函数**：提供了一种灵活的度量方式，可以控制函数的光滑程度，介于线性和平方指数之间。 5. **局部不变性**：某些协方差函数允许我们只关注输入点的相对距离，而不是绝对坐标，这在处理大规模数据时非常有用。除了介绍这些基本的协方差函数外，章节还可能讨论如何通过调整协方差函数的参数来适应不同的任务需求，以及如何通过正则化避免过拟合。此外，还会涉及协方差函数的组合，以构建更复杂的模型，适应更丰富的问题结构。在实际应用中，选择合适的协方差函数和优化其参数是高斯过程模型构建的重要步骤。通过合理选择和调整，高斯过程可以应用于回归、分类、优化、异常检测等多种机器学习任务。在《机械学习中的高斯过程》一书中，作者深入浅出地解释了这些概念，并提供了丰富的实例和理论分析，对于理解和掌握高斯过程有极大的帮助。

C. E. Rasmussen & C. K. I. Williams, Gaussian Processes for Machine Learning, the MIT Press, 2006,

ISBN 026218253X.



2006 Massachusetts Institute of Technology. www.GaussianProcess.org/gpml

4.2 Examples of Covariance Functions 83

rate of this decay of the power spectrum gives important information about

the smoothness of the associated stochastic process. For example it can deter-

mine the me an-square diﬀerentiability of the process (see section 4.3 for further

details).

If the covariance function is isotropic (so that it is a function of r, where

r = |τ |) then it can be shown that S(s) is a function of s , |s| only [Adler,

1981, Theorem 2.5.2]. In this case the integrals in eq. (4.6) can be simpliﬁed

by changing to spherical polar coordinates and integrating out the angular

variables (see e.g. Bracewell, 1986, ch. 12) to obtain

k(r) =

2π

D/2−1

∞

S(s)J

D/2−1

(2πrs)s

D/2

ds, (4.7)

S(s) =

2π

D/2−1

∞

k(r)J

D/2−1

(2πrs)r

D/2

dr, (4.8)

where J

D/2−1

is a Bessel function of order D/2−1. Note that the dependence on

the dimensionality D in equation 4.7 means that the same isotropic functional

form of the spectral density can give rise to diﬀerent isotropic covariance func-

tions in diﬀerent dimensions. Similarly, if we start with a particular isotropic

covariance function k(r) the form of spectral density will in general depend on

D (see, e.g. the Mat´ern class spectral density given in eq. (4.15)) and in fact

k(r) may not be valid for all D. A necessary condition for the spectral density

to exist is that

D−1

|k(r)|dr < ∞; see Stein [1999, sec. 2.10] for more details.

We now give s ome examples of commonly-used isotropic covariance func-

tions. The covariance functions are given in a normalized form where k(0) = 1;

we can multiply k by a (positive) constant σ

to get any desired process vari-

ance.

Squared Exponential Covariance Function

The squared exponential (SE) covariance function has already been introduced squared exponential

in chapter 2, eq. (2.16) and has the form

(r) = exp



−



, (4.9)

with parameter ` deﬁning the characteristic length-scale. Using eq. (4.3) we characteristic

length-scale

see that the mean number of level-zero upcrossings for a SE process in 1-

d is (2π`)

−1

, which conﬁrms the rˆole of ` as a length-scale. This covari-

ance function is inﬁnitely diﬀerentiable, which means that the GP with this

covariance function has mean square derivatives of all orders, and is thus

very smooth. The spectral density of the SE covariance function is S(s) =

(2π`

)

D/2

exp(−2π

). Stein [1999] argues that such strong smoothness

assumptions are unrealistic for modelling many physical processes, and rec-

ommends the Mat´ern class (see below). However, the squared exponential is

probably the most widely-used kernel within the ke rnel machines ﬁeld.

C. E. Rasmussen & C. K. I. Williams, Gaussian Processes for Machine Learning, the MIT Press, 2006,

ISBN 026218253X.



2006 Massachusetts Institute of Technology. www.GaussianProcess.org/gpml

84 Covariance Functions

The SE kernel is inﬁnitely divisible in that (k(r))

is a valid kernel for allinﬁnitely divisible

t > 0; the eﬀect of raising k to the power of t is simply to rescale `.

We now digress brieﬂy, to show that the squared exponential covariance

function can also be obtained by e xpanding the input x into a feature space

deﬁned by Gaussian-shaped basis functions centered densely in x-space. Forinﬁnite network

construction for SE

covariance function

simplicity of exposition we consider scalar inputs with basis functions

(x) = exp



−

(x − c)



, (4.10)

where c denotes the centre of the basis function. From sections 2.1 and 2.2 we

recall that w ith a Gaussian prior on the weights w ∼ N(0, σ

I), this gives rise

to a GP with covariance function

k(x

, x

) = σ

c=1

)φ

). (4.11)

Now, allowing an inﬁnite number of basis functions c entered everywhere on an

interval (and scaling down the variance of the prior on the weights with the

numbe r of basis functions) we obtain the limit

lim

N→∞

c=1

)φ

) = σ

max

min

)φ

)dc. (4.12)

Plugging in the Gaussian-shaped basis functions eq. (4.10) and letting the in-

tegration limits go to inﬁnity we obtain

k(x

, x

) = σ

∞

−∞

exp



−

− c)



exp



−

− c)



√

π`σ

exp



−

− x

)

√

2`)



(4.13)

which we recognize as a s quared exponential covariance function with a

√

times longer length-scale. The derivation is adapted from MacKay [1998]. It

is straightforward to generalize this construction to multivariate x. See also

eq. (4.30) for a similar construction where the centres of the basis functions are

sampled from a Gaussian distribution; the constructions are equivalent when

the variance of this Gaussian tends to inﬁnity.

The Mat´ern Class of Covariance Functions

The Mat´ern class of covariance functions is given byMat´ern class

Matern

(r) =

1−ν

Γ(ν)



√

2νr





√

2νr



, (4.14)

with positive parameters ν and `, where K

is a modiﬁed Bes se l function

[Abramowitz and Stegun, 1965, sec. 9.6]. This covariance function has a spectral

density

S(s) =

D/2

Γ(ν + D/2)(2ν)

Γ(ν)`

2ν



2ν

+ 4π



−(ν+D/2)

(4.15)

剩余25页未读，继续阅读

PassionalP

粉丝: 0
资源: 3

高斯过程在机器学习中的应用探析

2006年MIT经典：Gaussian Processes在机器学习中的革新策略

高斯过程在机器学习中的应用探索

Gaussian Processes详解：机器学习中的监督学习与回归/分类

gaussian processes for machine learning

电子书 Gaussian Processes for Machine Learning

Gaussian Processes for Machine Learning机器学习的高斯过程

gaussian processes for machine learning python版本

如何在Python中使用Gaussian Processes for Machine Learning库实现高斯过程回归，并进行超参数优化？

Gaussian Processes：机器学习的理论与实践指南

高斯过程：机器学习中的理论与应用

最新资源