数据挖掘与机器学习原理与理论

5星 · 超过95%的资源需积分: 10 95 浏览量更新于2023-07-12 5 收藏 13.08MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

"Principles and Theory for Data Mining and Machine Learning" 是一本深入介绍数据挖掘和机器学习核心概念与理论的书籍。它涵盖了从经典的函数估计到非线性回归、分类、集成方法等多个主题，并专注于聚类、降维、变量选择和多重比较。书中的每个主题都基于最新的发展，提供了现代视角，强调了近年来的贡献。虽然侧重于概念和理论，但书中通过计算比较强化了主要观点，并包含了使用R包进行进一步计算的完整代码示例。这本书面向统计学、计算机科学和电气工程的研究生，假设读者具有坚实的本科统计学和数学基础，以及使用R包的能力。书中有大量探索性问题，以及许多计算示例，便于读者实践。作者团队包括Bertrand Clarke、Ernest Fokoué和Hao Helen Zhang，他们都是在统计学领域有影响力的学者，其中Hao Helen Zhang曾获得国家科学基金会的早期职业发展奖。书中的内容包括但不限于：数据挖掘的基本原理，如如何从大量数据中提取有价值的信息；机器学习的理论基础，如监督学习和无监督学习的方法；非线性回归的理论和应用，用于处理复杂关系的数据建模；分类技术，如决策树、支持向量机等，用于预测和数据分类；集成方法，如随机森林和AdaBoost，这些方法通过组合多个弱学习器构建强预测模型；聚类方法，如K-means、层次聚类等，用于发现数据的内在结构；降维技术，如主成分分析和奇异值分解，帮助减少特征空间的复杂性；变量选择，探讨如何在众多变量中选择最相关的；以及多重比较，用于处理多组比较的统计问题。此书不仅是研究生教材，也是研究人员的手册，帮助他们理解数据挖掘和机器学习的核心概念。书中详尽的证明和实际案例使得理论知识更易理解和应用。通过阅读和实践本书，读者将能够掌握数据挖掘和机器学习领域的关键技能，并具备解决实际问题的能力。

资源详情

资源推荐

2 1 Variability, Information, and Prediction

Machine learning refers to the use of formal structures (machines) to do inference

(learning). This includes what empirical scientists mean by model building – proposing

mathematical expressions that encapsulate the mechanism by which a physical process

gives rise to observations – but much else besides. In particular, it includes many tech-

niques that do not correspond to physical modeling, provided they process data into

information. Here, information usually means anything that helps reduce uncertainty.

So, for instance, a posterior distribution represents “information” or is a “learner” be-

cause it reduces the uncertainty about a parameter.

The fusion of statistics, computer science, electrical engineering, and database man-

agement with new questions led to a new appreciation of sources of errors. In narrow

parametric settings, increasing the sample size gives smaller standard errors. However,

if the model is wrong (and they all are), there comes a point in data gathering where

it is better to use some of your data to choose a new model rather than just to con-

tinue reﬁning an existing estimate. That is, once you admit model uncertainty, you can

have a smaller and smaller variance but your bias is constant. This is familiar from

decomposing a mean squared error into variance and bias components.

Extensions of this animate DMML. Shrinkage methods (not the classical shrinkage,

but the shrinking of parameters to zero as in, say, penalized methods) represent a trade-

off among variable selection, parameter estimation, and sample size. The ideas become

trickier when one must select a basis as well. Just as there are well-known sums of

squares in ANOVA for quantifying the variability explained by different aspects of

the model, so will there be an extra variability corresponding to basis selection. In

addition, if one averages models, as in stacking or Bayes model averaging, extra layers

of variability (from the model weights and model list) must be addressed. Clearly,

good inference requires trade-offs among the biases and variances from each level of

modeling. It may be better, for instance, to “stack” a small collection of shrinkage-

derived models than to estimate the parameters in a single huge model.

Among the sources of variability that must be balanced – random error, parameter

uncertainty and bias, model uncertainty or misspeciﬁcation, model class uncertainty,

generalization error – there is one that stands out: model uncertainty. In the conven-

tional paradigm with ﬁxed parametric models, there is no model uncertainty; only

parameter uncertainty remains. In conventional nonparametrics, there is only model

uncertainty; there is no parameter, and the model class is so large it is sure to con-

tain the true model. DMML is between these two extremes: The model class is rich

beyond parametrization, and may contain the true model in a limiting sense, but the

true model cannot be assumed to have the form the model class deﬁnes. Thus, there

are many parameters, leading to larger standard errors, but when these standard errors

are evaluated within the model, they are invalid: The adequacy of the model cannot be

assumed, so the standard error of a parameter is about a value that may not be mean-

ingful. It is in these high-variability settings in the mid-range of uncertainty (between

parametric and nonparametric) that dealing with model uncertainty carefully usually

becomes the dominant issue which can only be tested by predictive criteria.

There are other perspectives on DMML that exist, such as rule mining, fuzzy learning,

observational studies, and computational learning theory. To an extent, these can be

regarded as elaborations or variations of aspects of the perspective presented here,

1 Variability, Information, and Prediction 3

although advocates of those views might regard that as inadequate. However, no book

can cover everything and all perspectives. Details on alternative perspectives to the one

perspective presented here can be found in many good texts.

Before turning to an intuitive discussion of several major ideas that will recur through-

out this monograph, there is an apparent paradox to note: Despite the novelty ascribed

to DMML, many of the topics covered here have been studied for decades. Most of

the core ideas and techniques have precedents from before 1990. The slight paradox is

resolved by noting that what is at issue is the novel, unexpected way so many ideas,

new and old, have been recombined to provide a new, general perspective dramatically

extending the conventional framework epitomized by, say, Lehmann’s books.

1.0.1 The Curse of Dimensionality

Given that model uncertainty is the key issue, how can it be measured? One crude

way is through dimension. The problem is that high model uncertainty, especially of

the sort central to DMML, rarely corresponds to a model class that permits a ﬁnite-

dimensional parametrization. On the other hand, some model classes, such as neural

nets, can approximate sets of functions that have an interior in a limiting sense and

admit natural ﬁnite-dimensional subsets giving arbitrarily good approximations. This

is the intermediate tranche between ﬁnite-dimensional and genuinely nonparametric

models: The members of the model class can be represented as limiting forms of an

unusually ﬂexible parametrized family, the elements of which give good, natural ap-

proximations. Often the class has a nonvoid interior.

In this context, the real dimension of a model is ﬁnite but the dimension of the model

space is not bounded. The situation is often summarized by the phrase the Curse of Di-

mensionality. This phrase was ﬁrst used by Bellman (1961), in the context of approx-

imation theory, to signify the fact that estimation difﬁculty not only increases with

dimension – which is no surprise – but can increase superlinearly. The result is that

difﬁculty outstrips conventional data gathering even for what one would expect were

relatively benign dimensions. A heuristic way to look at this is to think of real functions

of x,ofy, and of the pair (x, y). Real functions f , g of a single variable represent only

a vanishingly small fraction of the functions k of (x, y). Indeed, they can be embedded

by writing k(x, y)= f (x)+g(y). Estimating an arbitrary function of two variables is

more than twice as hard as estimating two arbitrary functions of one variable.

An extreme case of the Curse of Dimensionality occurs in the “large p,smalln”

problem in general regression contexts. Here, p customarily denotes the dimension

of the space of variables, and n denotes the sample size. A collection of such data is

1,i

,..., x

p,i

) for i = 1,...n. Gathering the explanatory variables, the x

i, j

s, into an

n×p matrix X in which the ith row is (x

1,i

,..., x

p,i

) means that X is short and fat when

p >> n. Conventionally, design matrices are tall and skinny, n >> p, so there is a rel-

atively high ratio n/p of data to the number of inferences. The short, fat data problem

occurs when n/p << 1, so that the parameters cannot be estimated directly at all, much

4 1 Variability, Information, and Prediction

less well. These problems need some kind of auxiliary principle, such as shrinkage or

other constraints, just to make solutions exist.

The ﬁnite-dimensional parametric case and the truly nonparametric case for regres-

sion are settings in which it is convenient to discuss some of the recurrent issues in

the treatments here. It will be seen that the Curse applies in regression, but the Curse

itself is more general, applying to classiﬁcation, and to nearly all other aspects of mul-

tivariate inference. As noted, traditional analysis avoids the issue by making strong

model assumptions, such as linearity and normality, to get ﬁnite-dimensional behav-

ior or by using distribution-free procedures, and being fully nonparametric. However,

the set of practical problems for which these circumventions are appropriate is small,

and modern applied statisticians frequently use computer-intensive techniques on the

intermediate tranche that are designed to minimize the impact of the Curse.

1.0.2 The Two Extremes

Multiple linear regression starts with n observations of the form (Y

) and then makes

the strong modeling assumption that the response Y

is related to the vector of explana-

tory variables X

=(X

1,i

,..., X

p,i

) by

1,i

+ ...

p,i

where each random error

is (usually) an independent draw from a normal distribu-

tion with mean zero and ﬁxed but unknown variance. More generally, the

saretaken

as symmetric, unimodal, and independent. The X

s can be random, or, more com-

monly, chosen by the experimenter and hence deterministic. In the chapters to follow,

instances of this setting will recur several times under various extra conditions.

In contrast, nonparametric regression assumes that the response variable is related to

the vector of explanatory variables by

= f (X

where f is some smooth function. The assumptions about the error may be the same

as for linear regression, but people tend to put less emphasis on the error structure

than on the uncertainty in estimates

f of f . This is reasonable because, outside of

large departures from independent, symmetric, unimodal

s, the dominant source of

uncertainty comes from estimating f . This setting will recur several times as well;

Chapter 2, for instance, is devoted to it.

Smoothness of f is central: For several nonparametric methods, it is the smoothness

assumptions that make theorems ensuring good behavior (consistency, for instance) of

regression estimators

f of f possible. For instance, kernel methods often assume f is

in a Sobolev space, meaning f and a ﬁxed number, say s, of its derivatives lie in a

Hilbert space, say L

(

), where the open set

⊂ R

is the domain of f .

1.1 Perspectives on the Curse 5

Other methods, like splines for instance, weaken these conditions by allowing f to be

piecewise continuous, so that it is differentiable between prespeciﬁed pairs of points,

called knots. A third approach penalizes the roughness of the ﬁtted function, so that the

data help determine how wiggly the estimate of f should be. Most of these methods

include a “bandwidth” parameter, often estimated by cross-validation (to be discussed

shortly). The bandwidth parameter is like a resolution deﬁning the scale on which

solutions should be valid. A ﬁner-scale, smaller bandwidth suggests high concern with

very local behavior of f ; a large-scale, higher bandwidth suggests one will have to be

satisﬁed, usually grudgingly, with less information on the detailed behavior of f .

Between these two extremes lies the intermediate tranche, where most of the action in

DMML is. The intermediate tranche is where the ﬁnite-dimensional methods confront

the Curse of Dimensionality on their way to achieving good approximations to the

nonparametric setting.

1.1 Perspectives on the Curse

Since almost all ﬁnite-dimensional methods break down as the dimension p of X

increases, it’s worth looking at several senses in which the breakdown occurs. This

will reveal impediments that methods must overcome. In the context of regression

analysis under the squared error loss, the formal statement of the Curse is:

• The mean integrated squared error of ﬁts increases faster than linearly in p.

The central reason is that, as the dimension increases, the amount of extra room in the

higher-dimensional space and the ﬂexibility of large function classes is dramatically

more than experience with linear models suggests.

For intuition, however, note that there are three nearly equivalent informal descriptions

of the Curse of Dimensionality:

• In high dimensions, all data sets are too sparse.

• In high dimensions, the number of possible models to consider increases superex-

ponentially in p.

• In high dimensions, all data sets show multicollinearity (or concurvity , which is

the generalization that arises in nonparametric regression).

In addition to these near equivalences, as p increases, the effect of error terms tends

to increase and the potential for spurious correlations among the explanatory variables

increases. This section discusses these issues in turn.

These issues may not sound very serious, but they are. In fact, scaling up most pro-

cedures highlights unforeseen weaknesses in them. To dramatize the effect of scaling

from two to three dimensions, recall the high school physics question: What’s the ﬁrst

thing that would happen if a spider kept all its proportions the same but was sud-

denly 10 feet tall? Answer: Its legs would break. The increase in volume in its body

6 1 Variability, Information, and Prediction

(and hence weight) is much greater than the increase in cross-sectional area (and hence

strength) of its legs. That’s the Curse.

1.1.1 Sparsity

Nonparametric regression uses the data to ﬁt local features of the function f inaﬂexi-

ble way. If there are not enough observations in a neighborhood of some point x

x, then

it is hard to decide what f (x

x) should be. It is possible that f has a bump at x

x, or a dip,

some kind of saddlepoint feature, or that f is just smoothly increasing or decreasing at

x. The difﬁculty is that, as p increases, the amount of local data goes to zero.

This is seen heuristically by noting that the volume of a p-dimensional ball of radius

r goes to zero as p increases. This means that the volume of the set centered at x

x in

which a data point x

must lie in order to provide information about f (x

x) has fewer and

fewer points per unit volume as p increases.

This slightly surprising fact follows from a Stirling’s approximation argument. Recall

the formula for the volume of a ball of radius r in p dimensions:

(p)=

p/2

(p/2 + 1)

. (1.1.1)

When p is even, p = 2k for some k. So,

lnV

(p)=k ln(

) −ln(k!)

since

(k + 1)=k!. Stirling’s formula gives k! ≈

√

k+1/2

−k

. So, (1.1.1) becomes

lnV

(p)=−

ln(2

) −

lnk +k[1 + ln(

)] −k lnk.

The last term dominates and goes to −∞ for ﬁxed r.Ifp = 2k + 1, one again gets

(p) → 0. The argument can be extended by writing

(p/2 + 1)=

((k + 1)+1/2)

and using bounds to control the extra “1/2”. As p increases, the volume goes to zero

for any r. By contrast, the volume of a cuboid of side length r is r

, which goes to 0,

1, or ∞ depending on r < 1, r = 1, or r > 1. In addition, the ratio of the volume of the

p-dimensional ball of radius r to the volume of the cuboid of side length r typically

goes to zero as p gets large.

Therefore, if the x

x values are uniformly distributed on the unit hypercube, the expected

number of observations in any small ball goes to zero. If the data are not uniformly dis-

tributed, then the typical density will be even more sparse in most of the domain, if a

little less sparse on a speciﬁc region. Without extreme concentration in that speciﬁc

region – concentration on a ﬁnite-dimensional hypersurface for instance – the increase

in dimension will continue to overwhelm the data that accumulate there, too. Essen-

tially, outside of degenerate cases, for any ﬁxed sample size n, there will be too few

data points in regions to allow accurate estimation of f .

剩余791页未读，继续阅读

Zylinkultamyyrä

粉丝: 10
资源: 103

会员权益专享

数据挖掘与机器学习原理与理论

Principles And Theory For Data Mining And Machine Learning

Feature Engineering for Machine Learning_Principles and Techniques

数据挖掘与机器学习原理与理论

What are the basic principles for using limited sample data to learn a learning machine and evaluate its performance?

Combining Prior Knowledge and Data for Robust Controller Design

integer quantization for deep learning inference: principles and empirical e

Multi-objective Evolutionary Federated Learning

pulse width modulation for power converters principles and practice

pulse width modulation for power converters: principles and practice 资源

principles and practice preason

principles of adaptive filters and self-learning systems

please introduce a number of OOP features (e.g. encapsulation) and principles (e.g. single responsibility principle).

贝叶斯网络在哪本书里有讲解

computational fluid dynamics - principles and applications - j. blazek.pdf

operating systems principles and practice

用英文回答人工智能的影响

principles of robot motion: theory algorithms and implementation. mit pres

patterns, principles, and practices of domain-driven design mobi pdf

写一篇对市场营销理论的自己的理解和它的应用

elctrical engineerinig : principles and applications

会员权益专享

最新资源