"数据挖掘与机器学习原理与理论：Springer统计学丛书"

需积分: 5 183 浏览量更新于2023-12-20 收藏 105.22MB PDF 举报

这段文字介绍了一本关于数据挖掘与机器学习原理与理论的著作，属于Springer Series in Statistics丛书的一部分。该著作的作者是Bertrand Clarke和Ernest Fokou´e，书中还有Hao Helen Zhang的参与。本书的主要内容涵盖了数据挖掘与机器学习的基本原理和理论。书籍的地址是在迈阿密大学的Bertrand Clarke和Rochester理工大学的Ernest Fokou´e所在单位。值得一提的是，该书是由Springer出版的，Springer是世界著名的科技研究出版机构，因此书籍的质量和学术价值非常可靠。本书也是该丛书中的一部分，该丛书的顾问包括了一些统计学领域的知名专家。作为一本关于数据挖掘与机器学习的著作，这本书涉及了当今信息技术领域中最为热门和重要的内容。随着大数据时代的到来，利用数据挖掘和机器学习技术来发现数据中的规律和信息已经成为了一种重要的手段。而这本书则系统地介绍了这些技术的理论基础和原则。对于从事数据科学、人工智能领域的学者和工程师来说，该书无疑具有非常重要的参考价值。书中所涉及的内容也将有助于读者更深入地理解数据挖掘和机器学习的内在原理，为他们在相关领域的研究和工作提供有力的支持。除了讨论这些技术的原理外，该书还着重于理论方面的阐述。这使得本书不仅是一本应用技术的教科书，更是一本系统而全面地介绍了数据挖掘与机器学习理论的学术著作。理论与实践相结合，这是本书的一个显著特点。通过深入地理解和掌握其中的理论知识，读者可以更好地应用这些技术到实际工作中，并更好地解决实际问题。这种理论和实践相结合的特点也使得本书具有了跨学科的特点，不仅对于计算机科学和信息学领域的学生和研究人员有益，对于统计学等其他相关领域的学者也是具有启发意义的。总体来说，这本书以其系统性、全面性和学术性，成为了一部不可多得的学术著作。对于计算机科学、信息学、统计学等领域的学者和学生来说，这是一本不可多得的宝贵之作。希望这本书的出版能够为相关领域的学术研究和实际应用提供有力的支持，也希望这本书能够为更多对数据挖掘与机器学习感兴趣的人士打开知识的窗户。

2 1 Variability, Information, and Prediction

Machine learning refers to the use of formal structures (machines) to do inference

(learning). This includes what empirical scientists mean by model building – proposing

mathematical expressions that encapsulate the mechanism by which a physical process

gives rise to observations – but much else besides. In particular, it includes many tech-

niques that do not correspond to physical modeling, provided they process data into

information. Here, information usually means anything that helps reduce uncertainty.

So, for instance, a posterior distribution represents “information” or is a “learner” be-

cause it reduces the uncertainty about a parameter.

The fusion of statistics, computer science, electrical engineering, and database man-

agement with new questions led to a new appreciation of sources of errors. In narrow

parametric settings, increasing the sample size gives smaller standard errors. However,

if the model is wrong (and they all are), there comes a point in data gathering where

it is better to use some of your data to choose a new model rather than just to con-

tinue reﬁning an existing estimate. That is, once you admit model uncertainty, you can

have a smaller and smaller variance but your bias is constant. This is familiar from

decomposing a mean squared error into variance and bias components.

Extensions of this animate DMML. Shrinkage methods (not the classical shrinkage,

but the shrinking of parameters to zero as in, say, penalized methods) represent a trade-

off among variable selection, parameter estimation, and sample size. The ideas become

trickier when one must select a basis as well. Just as there are well-known sums of

squares in ANOVA for quantifying the variability explained by different aspects of

the model, so will there be an extra variability corresponding to basis selection. In

addition, if one averages models, as in stacking or Bayes model averaging, extra layers

of variability (from the model weights and model list) must be addressed. Clearly,

good inference requires trade-offs among the biases and variances from each level of

modeling. It may be better, for instance, to “stack” a small collection of shrinkage-

derived models than to estimate the parameters in a single huge model.

Among the sources of variability that must be balanced – random error, parameter

uncertainty and bias, model uncertainty or misspeciﬁcation, model class uncertainty,

generalization error – there is one that stands out: model uncertainty. In the conven-

tional paradigm with ﬁxed parametric models, there is no model uncertainty; only

parameter uncertainty remains. In conventional nonparametrics, there is only model

uncertainty; there is no parameter, and the model class is so large it is sure to con-

tain the true model. DMML is between these two extremes: The model class is rich

beyond parametrization, and may contain the true model in a limiting sense, but the

true model cannot be assumed to have the form the model class deﬁnes. Thus, there

are many parameters, leading to larger standard errors, but when these standard errors

are evaluated within the model, they are invalid: The adequacy of the model cannot be

assumed, so the standard error of a parameter is about a value that may not be mean-

ingful. It is in these high-variability settings in the mid-range of uncertainty (between

parametric and nonparametric) that dealing with model uncertainty carefully usually

becomes the dominant issue which can only be tested by predictive criteria.

There are other perspectives on DMML that exist, such as rule mining, fuzzy learning,

observational studies, and computational learning theory. To an extent, these can be

regarded as elaborations or variations of aspects of the perspective presented here,

1 Variability, Information, and Prediction 3

although advocates of those views might regard that as inadequate. However, no book

can cover everything and all perspectives. Details on alternative perspectives to the one

perspective presented here can be found in many good texts.

Before turning to an intuitive discussion of several major ideas that will recur through-

out this monograph, there is an apparent paradox to note: Despite the novelty ascribed

to DMML, many of the topics covered here have been studied for decades. Most of

the core ideas and techniques have precedents from before 1990. The slight paradox is

resolved by noting that what is at issue is the novel, unexpected way so many ideas,

new and old, have been recombined to provide a new, general perspective dramatically

extending the conventional framework epitomized by, say, Lehmann’s books.

1.0.1 The Curse of Dimensionality

Given that model uncertainty is the key issue, how can it be measured? One crude

way is through dimension. The problem is that high model uncertainty, especially of

the sort central to DMML, rarely corresponds to a model class that permits a ﬁnite-

dimensional parametrization. On the other hand, some model classes, such as neural

nets, can approximate sets of functions that have an interior in a limiting sense and

admit natural ﬁnite-dimensional subsets giving arbitrarily good approximations. This

is the intermediate tranche between ﬁnite-dimensional and genuinely nonparametric

models: The members of the model class can be represented as limiting forms of an

unusually ﬂexible parametrized family, the elements of which give good, natural ap-

proximations. Often the class has a nonvoid interior.

In this context, the real dimension of a model is ﬁnite but the dimension of the model

space is not bounded. The situation is often summarized by the phrase the Curse of Di-

mensionality. This phrase was ﬁrst used by Bellman (1961), in the context of approx-

imation theory, to signify the fact that estimation difﬁculty not only increases with

dimension – which is no surprise – but can increase superlinearly. The result is that

difﬁculty outstrips conventional data gathering even for what one would expect were

relatively benign dimensions. A heuristic way to look at this is to think of real functions

of x,ofy, and of the pair (x, y). Real functions f , g of a single variable represent only

a vanishingly small fraction of the functions k of (x, y). Indeed, they can be embedded

by writing k(x, y)= f (x)+g(y). Estimating an arbitrary function of two variables is

more than twice as hard as estimating two arbitrary functions of one variable.

An extreme case of the Curse of Dimensionality occurs in the “large p,smalln”

problem in general regression contexts. Here, p customarily denotes the dimension

of the space of variables, and n denotes the sample size. A collection of such data is

1,i

,..., x

p,i

) for i = 1,...n. Gathering the explanatory variables, the x

i, j

s, into an

n×p matrix X in which the ith row is (x

1,i

,..., x

p,i

) means that X is short and fat when

p >> n. Conventionally, design matrices are tall and skinny, n >> p, so there is a rel-

atively high ratio n/p of data to the number of inferences. The short, fat data problem

occurs when n/p << 1, so that the parameters cannot be estimated directly at all, much

4 1 Variability, Information, and Prediction

less well. These problems need some kind of auxiliary principle, such as shrinkage or

other constraints, just to make solutions exist.

The ﬁnite-dimensional parametric case and the truly nonparametric case for regres-

sion are settings in which it is convenient to discuss some of the recurrent issues in

the treatments here. It will be seen that the Curse applies in regression, but the Curse

itself is more general, applying to classiﬁcation, and to nearly all other aspects of mul-

tivariate inference. As noted, traditional analysis avoids the issue by making strong

model assumptions, such as linearity and normality, to get ﬁnite-dimensional behav-

ior or by using distribution-free procedures, and being fully nonparametric. However,

the set of practical problems for which these circumventions are appropriate is small,

and modern applied statisticians frequently use computer-intensive techniques on the

intermediate tranche that are designed to minimize the impact of the Curse.

1.0.2 The Two Extremes

Multiple linear regression starts with n observations of the form (Y

) and then makes

the strong modeling assumption that the response Y

is related to the vector of explana-

tory variables X

=(X

1,i

,..., X

p,i

) by

1,i

+ ...

p,i

where each random error

is (usually) an independent draw from a normal distribu-

tion with mean zero and ﬁxed but unknown variance. More generally, the

saretaken

as symmetric, unimodal, and independent. The X

s can be random, or, more com-

monly, chosen by the experimenter and hence deterministic. In the chapters to follow,

instances of this setting will recur several times under various extra conditions.

In contrast, nonparametric regression assumes that the response variable is related to

the vector of explanatory variables by

= f (X

where f is some smooth function. The assumptions about the error may be the same

as for linear regression, but people tend to put less emphasis on the error structure

than on the uncertainty in estimates

f of f . This is reasonable because, outside of

large departures from independent, symmetric, unimodal

s, the dominant source of

uncertainty comes from estimating f . This setting will recur several times as well;

Chapter 2, for instance, is devoted to it.

Smoothness of f is central: For several nonparametric methods, it is the smoothness

assumptions that make theorems ensuring good behavior (consistency, for instance) of

regression estimators

f of f possible. For instance, kernel methods often assume f is

in a Sobolev space, meaning f and a ﬁxed number, say s, of its derivatives lie in a

Hilbert space, say L

(

), where the open set

⊂ R

is the domain of f .

1.1 Perspectives on the Curse 5

Other methods, like splines for instance, weaken these conditions by allowing f to be

piecewise continuous, so that it is differentiable between prespeciﬁed pairs of points,

called knots. A third approach penalizes the roughness of the ﬁtted function, so that the

data help determine how wiggly the estimate of f should be. Most of these methods

include a “bandwidth” parameter, often estimated by cross-validation (to be discussed

shortly). The bandwidth parameter is like a resolution deﬁning the scale on which

solutions should be valid. A ﬁner-scale, smaller bandwidth suggests high concern with

very local behavior of f ; a large-scale, higher bandwidth suggests one will have to be

satisﬁed, usually grudgingly, with less information on the detailed behavior of f .

Between these two extremes lies the intermediate tranche, where most of the action in

DMML is. The intermediate tranche is where the ﬁnite-dimensional methods confront

the Curse of Dimensionality on their way to achieving good approximations to the

nonparametric setting.

1.1 Perspectives on the Curse

Since almost all ﬁnite-dimensional methods break down as the dimension p of X

increases, it’s worth looking at several senses in which the breakdown occurs. This

will reveal impediments that methods must overcome. In the context of regression

analysis under the squared error loss, the formal statement of the Curse is:

• The mean integrated squared error of ﬁts increases faster than linearly in p.

The central reason is that, as the dimension increases, the amount of extra room in the

higher-dimensional space and the ﬂexibility of large function classes is dramatically

more than experience with linear models suggests.

For intuition, however, note that there are three nearly equivalent informal descriptions

of the Curse of Dimensionality:

• In high dimensions, all data sets are too sparse.

• In high dimensions, the number of possible models to consider increases superex-

ponentially in p.

• In high dimensions, all data sets show multicollinearity (or concurvity , which is

the generalization that arises in nonparametric regression).

In addition to these near equivalences, as p increases, the effect of error terms tends

to increase and the potential for spurious correlations among the explanatory variables

increases. This section discusses these issues in turn.

These issues may not sound very serious, but they are. In fact, scaling up most pro-

cedures highlights unforeseen weaknesses in them. To dramatize the effect of scaling

from two to three dimensions, recall the high school physics question: What’s the ﬁrst

thing that would happen if a spider kept all its proportions the same but was sud-

denly 10 feet tall? Answer: Its legs would break. The increase in volume in its body

6 1 Variability, Information, and Prediction

(and hence weight) is much greater than the increase in cross-sectional area (and hence

strength) of its legs. That’s the Curse.

1.1.1 Sparsity

Nonparametric regression uses the data to ﬁt local features of the function f inaﬂexi-

ble way. If there are not enough observations in a neighborhood of some point x

x, then

it is hard to decide what f (x

x) should be. It is possible that f has a bump at x

x, or a dip,

some kind of saddlepoint feature, or that f is just smoothly increasing or decreasing at

x. The difﬁculty is that, as p increases, the amount of local data goes to zero.

This is seen heuristically by noting that the volume of a p-dimensional ball of radius

r goes to zero as p increases. This means that the volume of the set centered at x

x in

which a data point x

must lie in order to provide information about f (x

x) has fewer and

fewer points per unit volume as p increases.

This slightly surprising fact follows from a Stirling’s approximation argument. Recall

the formula for the volume of a ball of radius r in p dimensions:

(p)=

p/2

(p/2 + 1)

. (1.1.1)

When p is even, p = 2k for some k. So,

lnV

(p)=k ln(

) −ln(k!)

since

(k + 1)=k!. Stirling’s formula gives k! ≈

√

k+1/2

−k

. So, (1.1.1) becomes

lnV

(p)=−

ln(2

) −

lnk +k[1 + ln(

)] −k lnk.

The last term dominates and goes to −∞ for ﬁxed r.Ifp = 2k + 1, one again gets

(p) → 0. The argument can be extended by writing

(p/2 + 1)=

((k + 1)+1/2)

and using bounds to control the extra “1/2”. As p increases, the volume goes to zero

for any r. By contrast, the volume of a cuboid of side length r is r

, which goes to 0,

1, or ∞ depending on r < 1, r = 1, or r > 1. In addition, the ratio of the volume of the

p-dimensional ball of radius r to the volume of the cuboid of side length r typically

goes to zero as p gets large.

Therefore, if the x

x values are uniformly distributed on the unit hypercube, the expected

number of observations in any small ball goes to zero. If the data are not uniformly dis-

tributed, then the typical density will be even more sparse in most of the domain, if a

little less sparse on a speciﬁc region. Without extreme concentration in that speciﬁc

region – concentration on a ﬁnite-dimensional hypersurface for instance – the increase

in dimension will continue to overwhelm the data that accumulate there, too. Essen-

tially, outside of degenerate cases, for any ﬁxed sample size n, there will be too few

data points in regions to allow accurate estimation of f .

剩余791页未读，继续阅读

承让@

粉丝: 8
资源: 380

"数据挖掘与机器学习原理与理论：Springer统计学丛书"

数据挖掘 机器学习原理与SPSS Clementine应用宝典 第4章 数据选择.pdf

数据挖掘 机器学习原理与SPSS Clementine应用宝典 第18章 SPSS Clementine基础.ppt

数据挖掘 机器学习原理与SPSS Clementine应用宝典 第10章 关联规则 共65页.ppt

数据挖掘涉及什么知识或工具

usct 机器学习与知识发现考试

csdn 叶齐祥 机器学习

数据挖掘期末试卷csdn

python 数据挖掘 大作业

推荐一个Python数据挖掘课程设计

最新资源

数据挖掘机器学习原理与SPSS Clementine应用宝典第4章数据选择.pdf

数据挖掘机器学习原理与SPSS Clementine应用宝典第18章 SPSS Clementine基础.ppt

数据挖掘机器学习原理与SPSS Clementine应用宝典第10章关联规则共65页.ppt

csdn 叶齐祥机器学习

python 数据挖掘大作业