信息论基础与机器学习应用

需积分: 21 23 浏览量更新于2024-07-20 1 收藏 670KB PDF 举报

身份认证购VIP最低享 7 折!

30元优惠券

"信息论与机器学习是紧密相关的领域，信息论提供了一套数学工具来量化和理解数据中的信息，而机器学习则依赖于这些工具进行有效的数据建模和预测。Massimiliano Tomassoli的文档详细介绍了信息论的基本概念，并探讨了它们在机器学习中的应用。 1. **信息理论基础** - **平均值或期望值**：在概率论中，平均值或期望值是衡量随机变量平均结果的统计量。 - **熵**：熵是度量一个随机变量不确定性的量，越高表示不确定性越大。 - **熵与编码**：熵在数据压缩和编码中起着关键作用，因为它决定了最小可能的平均编码长度。 - **对数与概率**：在信息论中，对数常用于转换概率，便于计算。 - **Jensen's不等式**：Jensen's不等式涉及到函数的凸性，对于凸函数，其在概率分布上的期望值至少等于该函数在期望值处的值。 2. **最大熵原理** - 最大熵原理指出，在所有满足已知约束的分布中，熵最大的分布最能代表我们对系统的无知。 3. **指定分布** - 在机器学习中，我们需要根据数据来确定概率分布。 4. **链式法则** - 链式法则在计算多变量联合概率时非常有用，允许我们将复杂的概率问题分解为更简单的部分。 5. **交叉熵** - 交叉熵是评估两个概率分布之间差异的度量，常用于损失函数。 6. **Kullback-Leibler散度** - K-L散度是衡量一个概率分布P相对于另一个分布Q的“距离”，在信息论和机器学习中用于模型的比较和优化。 7. **互信息** - 互信息测量两个随机变量之间的相关性，它不是对称的，且可以为零，即使变量之间有相关性。 - 互信息与相关性的区别在于，互信息考虑了变量间的依赖结构，而不仅仅是线性关系。 - 在线性回归中，互信息可以用来衡量自变量对因变量的解释力。 8. **信息论的集合论视角** - 集合论视角提供了更深入的理解，如包含-排除原理、交集的基数，以及多重互信息和总相关性的关系。 9. **信息论在机器学习中的应用** - **损失函数**：从K-L散度到交叉熵再到对数似然，这些信息论概念在定义机器学习模型的优化目标中起到关键作用。 - **特征选择**：信息论可以用于选择最有信息量的特征，以提高模型的性能。信息论不仅为机器学习提供了理论基础，还在实际的建模和预测任务中扮演着核心角色，如损失函数设计、特征选择和模型评估等方面。通过深入理解这些概念，可以更有效地构建和优化机器学习模型。

资源详情

资源推荐

3 Entropy

Entropy is a function which assigns single real numbers to ﬁnite discrete dis-

tributions. Some people refer directly to the distributions and others to the

random variables associated with them. We’ll be using whatever notation is

more convenient for what we want to say. Be ﬂexible and learn not to be thrown

oﬀ by diﬀerent notations!

In what follows, X, Y, Z are always discrete random variables, p, q, r are

ﬁnite discrete distributions, and x, y, z are values taken by the variables X, Y, Z,

respectively.

Deﬁnition 8. Self-information

If X follows distribution p, the self-information of x is deﬁned as

I(x) = log

p(x)

= − log

p(x).

The formula above says that values with high probability have low self-

information, whereas values with low probability have high self-information.

We’ll see later why this makes sense.

Note that I measures the amount of information. We’ll often say just “in-

formation” instead of “amount of information”.

Deﬁnition 9. Entropy

The entropy of X is deﬁned as

H(X) = E

[I(X)] = −

p(x) log

p(x).

The entropy is the expected amount of information contained in a random

variable. A variable may assume values x

, x

, . . . , x

and each value may have

a diﬀerent self-information so it makes sense to summarize the information con-

tained in all the values with the mean.

As always, note that we didn’t write E

[I(x)] with a lowercase x. While

I(x) is just a real number, I(X) is a random variable obtained by applying the

function I to the random variable X.

4 Entropy and Codes

Consider the following scenario. We have a random variable X with distribution

p. We take samples x

, x

, . . . from p and send these values through a channel.

We send the samples as we generate them so this process may go on forever.

To send the samples, we must encode them some way. We can deﬁne a code,

i.e. a function C : X → Σ

∗

which maps values x to words over an alphabet Σ,

which is just a set of arbitrary symbols. The asterisk is the Kleene star, which

takes a set of symbols and returns the set of all the words (of ﬁnite non-negative

length) over those symbols. For instance,

{0, 1}

∗

= {, 0, 1, 00, 01, 10, 11, 000, 001, . . .}

is the set of all the ﬁnite binary strings, including the empty string (denoted by

).

The extension of C is deﬁned as follows:

∗

, x

, . . . , x

) = C(x

)C(x

) · · · C(x

)

where the juxtaposition indicates concatenation of strings. Basically, we encode

sequences of samples x

by concatenating the encoded samples. We say that C

is uniquely decodable if C

∗

is injective. Note that if C

∗

is not injective, then

there are two diﬀerent sequences of samples which result in the same encoded

string and thus, given the encoded string, we can’t recover the original sequence

of samples (we can recover both sequences but which one is the right one?).

Theorem 1. Shannon’s Source Code Theorem

Let X be a discrete random variable and C : X → Σ

∗

a uniquely decodable

code. If C is optimal, i.e. it minimizes E

[length(C(X))], then

H(X)

log

|Σ|

≤ E

[length(C(X))] <

H(X)

log

|Σ|

+ 1.

To understand theorem 1 better, note that if we choose Σ = {0, 1}, i.e. we

encode the symbols as binary strings, then the inequality simpliﬁes to

H(X) ≤ E

[length(C(X))] < H(X) + 1.

We can also be more explicit:

H(X) ≤ min

[length(C(X))] < H(X) + 1.

So, we can’t do better than H(X): no matter which code we choose, the

expected length of the encoded symbols will be at least H(X). Note, though,

that if the code is optimal then we can’t do worse than H(X) + 1. But where

does that +1 come from? Before ﬁnding that out, we must observe a few things.

You should remember that H(X) is the expected self-information, so we can

rewrite the formula above as

[I(X)] ≤ min

[length(C(X))] < E

[I(X)] + 1.

This suggests that I(X) has something to do with length(C(X)). In fact, I(x)

measures the information “contained” in x by counting the number of bits an

optimal code uses to represent x. As we observed before, the deﬁnition I(x) =

log

p(x)

suggests that an optimal code will use short strings for frequent samples

and long strings for rare samples. (As a side note, observe that if X was a

continuous random variable then any sample x would be inﬁnitely rare and

our reasoning would fall apart!) This explains, at least in part, why the self-

information and the entropy contain a logarithm.

剩余48页未读，继续阅读

EricAn

粉丝: 2589
资源: 86

信息论基础与机器学习应用

脑智术语建议-胡包钢NLPR

格数致知-如何从计算层面发展类脑智能_胡包钢NLPR

信息论-香农.pdf

信息论与机器学习的联系

信息论与机器学习与人工智能

信息论与机器学习的关系

信息论在机器学习中的应用

机器学习之矩阵论 csdn 网盘

生成一篇机器学习的发展与现状的论文

信息论 推理与算法pdf

因果推断与机器学习 pdf

机器学习数学理论 pdf

与机器学习有关的熵算法

信息论基础知识的应用

关于信息论的五个参考文献

信息论在人工智能中的运用

机器学习常用的与熵有关的算法

信息论基础英文版pdf

人工智能 机器学习 sci期刊

信息论基础理论与应用课后答案csdn

最新资源

信息论推理与算法pdf

人工智能机器学习 sci期刊