机器学习统计基础详解

需积分: 9 82 浏览量更新于2024-07-21 收藏 8.57MB PDF 举报

《统计机器学习基础手册》由Gianluca Bontempi撰写，出自布鲁塞尔自由大学计算机科学系的机器学习小组。这本书深入探讨了统计与机器学习之间的关键联系，旨在为理解这两种领域提供坚实的理论基础。以下是该书的部分核心知识点： 1. **引言**：章节介绍了统计机器学习的基础概念，包括符号定义和背景，强调了概率模型在不确定性建模中的核心作用。 2. **概率基础**： - **随机模型的不确定性**：讨论了概率的几种定义方法，包括公理化定义（基于一组假设条件），对称性定义，以及频率主义观点（通过大量观察结果的频率来确定概率）。 - **大数定律**：阐述了在大量重复实验中，某些事件发生的频率趋向于理论概率的现象。 - **独立性和条件概率**：讲解了两个或多个事件之间的相互关系以及如何计算条件概率。 - **联合实验与全概率定理和贝叶斯定理**：介绍如何处理涉及多个事件的复杂情况，包括全概率公式和贝叶斯法则，这些是预测和决策过程的重要工具。 - **联合与边缘概率阵列**：通过矩阵形式展示概率的组合，便于理解和分析多变量系统。 3. **随机变量**：区分了离散随机变量和连续随机变量，分别讨论它们的概率分布、期望值、方差、标准差等特性。 - **离散随机变量**：讲解参数化概率函数，如二项分布、几何分布等，并强调了计算其统计量的方法。 - **连续随机变量**：介绍均值、方差和矩的概念，以及如何处理连续分布如正态分布。 4. **联合概率**：重点讨论了联合概率分布及其与边际概率和条件概率的关系，这对于理解数据依赖和变量间关系至关重要。通过这本手册，读者能够建立起扎实的统计学基础，进而将这些理论应用于机器学习模型的设计、评估和优化过程中。理解这些概念对于从事数据科学、人工智能领域的专业人士来说是至关重要的，因为它们构成了模型训练和决策分析的核心组成部分。

16 CHAPTER 1. INTRODUCTION

Probability Theory notation

Ω Set of possible outcomes.

ω Outcome (or elementary event).

{E} Set of possible events.

E Event.

Prob {E} Probability of the event E.

(Ω, {E}, Prob {·}) Probabilistic model of an experiment.

Z Domain of the random variable z.

P (z) Probability distribution of a discrete random variable z. Also P

(z).

F (z) = Prob {z ≤ z} Distribution function of a continuous random variable z. Also F

(z).

p(z) Probability density of a continuous r.v.. Also p

(z).

E[z] Expected value of the random variable z.

[z] =

z(x, y)p(x)dx Expected value of the random variable z averaged over x.

Var [z] Variance of the random variable z.

L(θ) Likelihood of a parameter θ.

l(θ) Log-Likelihood of a parameter θ.

emp

(θ) Empirical Log-likelihood of a parameter θ.

Learning Theory notation

x Multidimensional input variable.

X ⊂ <

Input space.

y Multidimensional output variable.

Y ⊂ < Output space.

realization of the random vector x.

f(x) Target regression function.

w Random noise variable.

= hx

, y

i Input-output sample: i

case in training set.

N Number of available samples.

= {z

, z

, . . . , z

} Training set.

Λ Class of hypothesis.

α Hypothesis parameter vector.

h(x, α) Hypothesis function.

Hypothesis class of complexity s.

L(y, f(x, α)) Loss function.

R(α) Functional risk.

arg min

α∈Λ

R(α).

emp

(α) Empirical functional risk.

arg min R

emp

(α).

Mean integrated squared error (MISE).

l Number of folds in cross-validation.

Cross-validation estimate of G

loo

Leave-one-out estimate of G

Number of samples used for training in cross-validation.

Number of samples used for test in cross-validation.

i = 1, . . . , l Parameter which minimizes the empirical risk of D

(i)

Training set with the i

sample set aside.

N(i)

Parameter which minimizes the empirical risk of D

(i)

boot

Bootstrap estimate of G

∗

Bootstrap training set of size N generated by D

with replacement.

Number of bootstrap samples.

Chapter 2

Foundations of probability

Probability theory is the discipline concerned with the study of uncertain (or ran-

dom) phenomena and probability is the mathematical language adopted for quan-

tifying uncertainty. Such phenomena, although not predictable in a deterministic

fashion, may present some regularities and consequently be described mathemati-

cally by idealized probabilistic models. These models consist of a list of all possi-

ble outcomes together with the respective probabilities. The theory of probability

makes possible to infer from these models the patterns of future behaviour.

This chapter presents the basic notions of probability which serves as a necessary

background to understand the statistical aspects of machine learning. We ask the

reader to become acquainted with two aspects: the notion of random variable as

a compact representation of uncertain knowledge and the use of probability as an

eﬀective formal tool to manipulate and process uncertain information. In particular,

we suggest the reader give special attention to the notions of conditional and joint

probability. As we will see in the following, these two related notions are extensively

used by statistical modelling and machine learning to deﬁne the dependence and

the relationships between random variables.

2.1 The random model of uncertainty

We deﬁne a random experiment as any action or process which generates results or

observations which cannot be predicted with certainty. Uncertainty stems from the

existence of alternatives. In other words, each uncertain phenomenon is character-

ized by a multiplicity of possible conﬁgurations or outcomes. Weather is uncertain

since it can take multiple forms (e.g. sunny, rainy, cloudy,...). Other examples of

random experiments are tossing a coin, rolling dice, or measuring the time to reach

home.

A random experiment is then characterized by a sample space Ω that is a (ﬁnite

or inﬁnite) set of all the possible outcomes (or conﬁgurations) ω of the experiment.

The elements of the set Ω are called experimental outcomes or realizations. For

example, in the die experiment, Ω = {ω

, ω

, . . . , ω

} and ω

stands for the outcome

corresponding to getting the face with the number i. If ω is the outcome of a

measurement of some physical quantity, e.g. pressure, then we could have Ω = R

The representation of an uncertain phenomenon is the result of a modelling

activity and as such it is not necessarily unique. In other terms diﬀerent repre-

sentations of a random experiment are possible. In the die experiment, we could

deﬁne an alternative sample space made of two sole outcomes: numbers equal to

and diﬀerent from 1. Also we could be interested in representing the uncertainty of

two consecutive tosses. In that case the outcome would be the pair (ω

, ω

20 CHAPTER 2. FOUNDATIONS OF PROBABILITY

Uncertainty stems from variability. Each time we observe a random phenomenon,

we may observe diﬀerent outcomes. In the probability jargon, observing a random

phenomenon is interpreted as the realization of a random experiment. A single

performance of a random experiment is called a trial. This means that after each

trial we observe one outcome ω

∈ Ω.

A subset of experimental outcomes is called an event. Consider a trial which

generated the outcome ω

: we say that an event E occurred during the trial if the

set E contains the element ω

. For example, in the die experiment, an event is the

set of even values E = {ω

, ω

}. This means that when we observe the outcome

the event even number takes place.

An event composed of a single outcome, e.g. E = {ω

} is called an elementary

event.

Note that since events E are subsets, we can apply to them the terminology of

the set theory:

• Ω refers to the certain event i.e. the event that occurs in every trial.

• the notation

= {ω : ω /∈ E}

denotes the complement of E.

• the notation

∪ E

= {ω ∈ Ω : ω ∈ E

OR ω ∈ E

}

refers to the event that occurs when E

or E

or both occur.

• the notation

∩ E

= {ω ∈ Ω : ω ∈ E

AND ω ∈ E

}

refers to the event that occurs when both E

and E

occur.

• two events E

and E

are mutually exclusive or disjoints if

∩ E

= ∅

that is each time that E

occurs, E

does not occur.

• a partition of Ω is a set of disjoint sets E

, j = 1, . . . , n such that

∪

j=1

= Ω

• given an event E we deﬁne the indicator function of E by

(ω) =

(

1 if ω ∈ E

0 if ω /∈ E

(2.1.1)

Let us consider now the notion of class of events. An arbitrary collection of

subsets of Ω is not a class of events. We require that if E

and E

are events, the

same holds also for the intersection E

∩E

and the union E

∪E

. A set of events that

satisﬁes these conditions is called, in mathematical terms, a Borel ﬁeld [91]. We

will consider only Borel ﬁelds since we want to deal not only with the probabilities

of single events, but also with the probabilities of their unions and intersections.

剩余266页未读，继续阅读

Harveyyyyy

粉丝: 0
资源: 1

机器学习统计基础详解

Foundations_of_Data_Science_March 2019.pdf

machine learning ppt

MATLAB Gaussian Fitting in Machine Learning: Foundation of Constructing Predictive Models, Enhancing...

Pattern Recogintion and Machine Learning

MIT Optimization for Machine Learning.

Springer-Modern.Multivariate.Statistical.Techniques.Regression.classification.and.manifold.learning.(2008)

Unsupervised Learning by Probabilistic Latent Semantic Analysis

Learning.Quantitative.Finance.with.R.azw3

MATLAB Toolbox Detailed Explanation: Statistics and Machine Learning Toolbox

MATLAB Versions and Machine Learning: Advantages and Limitations, Exploring Different Versions

最新资源