限制域下核心稳定委员会：稀疏连续分布与Fenchel-Young损失研究

版权申诉

126 浏览量更新于2024-07-06 收藏 7.41MB PDF 举报

身份认证购VIP最低享 7 折!

30元优惠券

本文探讨了"限制域下的核心稳定委员会(Core-Stable Committees)"在机器学习领域的应用，特别是在处理稀疏连续分布与Fenchel-Young损失函数的情境下。研究者们，包括来自葡萄牙里斯本的Andr'e F.T. Martins、Marcos Treviso、Antônio Farinhas、Pedro M.Q. Aguiar、Mário A.T. Figueiredo、Mathieu Blondel（Google Research）以及Vlad Niculae（University of Amsterdam），共同合作，聚焦于Exponential Families（指数家族）在现代技术中的关键角色。 Exponential Families是概率论中的一个重要概念，它们是一类参数化概率分布，其概率密度函数的形式具有指数形式，这使得它们在统计学和机器学习中具有广泛的应用。这类分布的特点在于参数可以方便地表示为一个基函数的线性组合，且包含许多常见的概率模型，如高斯分布、伯努利分布等。 Fenchel-Young损失函数，也被称为Fenchel对偶或Hinge损失，是支持向量机（SVM）中最常用的损失函数之一。它在二分类问题中表现优异，特别是当特征空间是线性不可分时，通过最大化间隔来提高泛化能力。在稀疏连续分布的背景下，Fenchel-Young损失能够有效地处理数据的稀疏性，这对于大数据集和高维特征空间尤为重要，因为它有助于减少计算复杂性和内存需求。核心稳定委员会这一概念可能指的是在特定的受限环境中，如何构建一个稳定的决策或优化过程，它可能涉及算法的稳定性和鲁棒性分析。在限制域下，这意味着理论和实践上的挑战，可能包括如何在有限的数据或计算资源约束下，确保模型的性能和有效性。论文作者们的研究工作可能围绕以下几个方面展开： 1. **稀疏连续分布的特性分析**：研究如何利用Exponential Families的结构来处理稀疏数据，并探究这种分布如何影响模型的训练效率和性能。 2. **Fenchel-Young损失与核心稳定性**：探讨如何结合Fenchel-Young损失的特性，设计出在受限域下依然保持稳定性的算法，以提高模型的泛化能力和适应性。 3. **算法设计与优化**：提出核心稳定委员会的方法论，可能涉及到优化算法的设计，以及如何在有限计算资源下实现高效的学习。 4. **实验与评估**：通过实证研究验证理论分析，展示在特定限制条件下，核心稳定委员会策略的实际效果，可能包括比较与其他方法的性能和效率。 5. **应用前景**：讨论这些研究成果在实际机器学习任务中的潜在应用，比如推荐系统、图像识别或者自然语言处理等领域。这篇论文深入探讨了在受限领域内如何利用Exponential Families和Fenchel-Young损失来构建核心稳定委员会，以提升机器学习模型的性能和效率。这对于理解复杂数据集和优化计算策略具有重要的理论价值和实践意义。

资源详情

资源推荐

Sparse Continuous Distributions and Fenchel-Young Losses

2 0

Gaussian ( = 1)

0.01

0.03

0.10

0.30

2 0

Triweight ( = 4/3)

2 0

Biweight ( = 3/2)

2 0

Epanechnikov ( = 2)

0 2

Gaussian ( = 1)

0 2

Triweight ( = 4/3)

0 2

Biweight ( = 3/2)

0 2

Epanechnikov ( = 2)

Figure 4:

Density and samples of two-dimensional

-Gaussian random variables, for the

isotropic (top) and anisotropic cases (bottom). These distributions are obtained by

applying (Tsallis) Ω

-regularized prediction maps to quadratic scoring functions,

with

= 2

− β

. Shown are the original density (solid lines) and the density ﬁt to

samples by moment matching (dashed lines). All contour lines are at the same

absolute levels, and the complement of the support is shaded when appropriate.

Examples 1 and 2: Squared and absolute losses.

Let Ω be the Shannon-Boltzmann-

Gibbs negentropy. Let

, i.e., the distribution contains a single sample

. For the

Gaussian distribution, by identiﬁcation with

(3)

, we get Ω

∗

(

) =

(

) =

log

(

√

2π

) and

therefore

Ω

(f; p) =

(t − µ)

2σ

+ log(σ

√

2π).

Similarly, for the Laplace distribution, we get Ω

∗

(f) = A(f) = log(2b) and therefore

Ω

(f; p) =

|t − µ|

+ log(2b).

Example 3: KL divergence between two Gaussian distributions.

When Ω is

the Shannon-Boltzmann-Gibbs entropy, the Fenchel dual is the log-partition function

(3)

Ω

∗

(

) =

(

), and the Fenchel-Young loss recovers the Kullback-Leibler divergence. For

example, if

(

) =

(

;

µ,

Σ), and

(

) =

−

(

t − µ

)

−1

(

t − µ

), using the

expression for the entropy in

2, we obtain the well-known expression for the Kullback-

Leibler divergence between Gaussians:

Ω

(f; p) =

(µ − µ

)

−1

(µ − µ

) +



Tr(Σ

−1

Σ) − N + log

|Σ

|Σ|



. (6)

6, we will generalize this result for a class of elliptical distributions called

-Gaussian

distributions, for which we will derive a closed-form expression for the Fenchel-Young loss.

Sparse Continuous Distributions and Fenchel-Young Losses

4. Tsallis Regularizers and Deformed Exponential Families

We introduce in this section a broader set of regularizers Ω based on Tsallis entropies (Tsallis,

1988). Tsallis entropies are a generalization of Shannon-Boltzmann-Gibbs entropies which

are suitable to model several phenomena present in natural, artiﬁcial and social complex

systems (Lutz, 2003; Burlaga et al., 2005; Pickup et al., 2009; Adare et al., 2011, inter

alia) under the umbrella of nonextensive statistical mechanics (Abe and Okamoto, 2001),

a generalization of the Boltzmann-Gibbs theory. We will see that using these regularizers

in Deﬁnition 2 leads to “deformed exponential families,” which may correspond to sparse

density functions in the sense of Deﬁnition 1. This makes a bridge between the entmax

transformation, proposed for ﬁnite domains by Blondel et al. (2020) and Peters et al. (2019),

and new transformations which we will propose in §5 and §6 for the non-ﬁnite case.

4.1 Tsallis entropies

A central concept in Tsallis statistics is a generalization of the standard logarithm and

exponential functions, called

β-logarithm

log

≥0

→ R

(not to be confused with base-

logarithm), and β-exponential, exp

: R → R, deﬁned as follows:

log

(u) :=

(

1−β

−1

1−β

, β 6= 1

log u, β = 1;

exp

(u) :=

(

[1 + (1 − β)u]

1/(1−β)

, β 6= 1

exp u, β = 1.

(7)

Note that

lim

β→1

log

(

) =

log u

lim

β→1

exp

(

) =

exp u

, and

log

(

exp

(

)) =

for any

and

u ∈ R

. Another important concept, which we will use in the sequel, is that of

β-escort

distribution

(Tsallis, 1988): this is the distribution

˜p

obtained by applying the following

“sharpness” operator:

p(t) 7→ ˜p

(t) :=

p(t)

kpk

, where kpk

p(t

)

dν(t

). (8)

Note that we have

˜p

(

) =

(

β >

1 increases sharpness, whereas

β <

1 decreases it,

producing more uniform distributions. β = 0 results in a uniform distribution.

We thus have the following deﬁnition (Havrda and Charv´at, 1967; Tsallis, 1988):

Deﬁnition 4 (Tsallis negentropies.) For α ≥ 0, the α-Tsallis negentropy is:

Ω

(p) :=

[log

2−α

(p(t))] =

(

α(α−1)



p(t)

− 1



, α 6= 1,

p(t) log p(t), α = 1.

(9)

The family of Tsallis entropies is continuous in

, i.e.,

lim

α→1

Ω

(

) = Ω

(

), for any

p ∈ M

(

), with Ω

(

) recovering Shannon’s negentropy (see Appendix C for a proof).

Another notable case is Ω

(

) =

(

)

−

, the negative of which has several names,

e.g., Gini-Simpson index (Jost, 2006) or Rao’s quadratic entropy (Rao, 1982). We will come

back to the α = 2 case in §5.

This entropy is normally deﬁned up to a constant, often presented without the

factor. We use the

same deﬁnition as Blondel et al. (2020, §4.3) for convenience.

Sparse Continuous Distributions and Fenchel-Young Losses

4.2 Tsallis regularization: deformed exponential families and α-sparse families

For

α >

0, the Tsallis negentropy Ω

is strictly convex, hence it can be plugged as the

regularizer in Deﬁnition 2. The next proposition is a reformulation of a result due to Naudts

(2009) in the statistical physics literature; we include a proof in Appendix C. This result

provides an expression for the Ω

-regularized prediction map:

Proposition 4 (Distribution and normalizing function expressions)

For

α >

0 and f ∈ F, the Ω

-regularized prediction map has the following form:

ˆp

Ω

[f](t) = exp

2−α

(f(t) − A

(f)), (10)

where

exp

is deﬁned in

(7)

and

F → R

is a normalizing function (we write

p(t) ≡ ˆp

Ω

[f](t) for simplicity):

(f) =

1−α

p(t)

2−α

f(t)

p(t)

2−α

−

1 − α

It is interesting to contrast

(10)

with Boltzmann-Gibbs distributions

(3)

, which are recovered

as a limit case when

α →

1. One key aspect to note is that the (2

− α

)-exponential, for

α >

1, can return zero values. Therefore,

the distribution ˆp

Ω

[

]

in (10) might not

have full support, i.e., we may have supp

(

ˆp

Ω

[

])

( S.

In particular, it may be a

sparse density function

in the sense of Deﬁnition 1 (see Figure 3). This never happens

with Boltzmann-Gibbs distributions, which always have full support.

Relation to sparsemax and entmax.

Blondel et al. (2020) showed that, for ﬁnite

, Ω

-regularized prediction map (i.e., picking

= 2) is the

sparsemax

transformation,

ˆp

Ω

[

] =

sparsemax

(

) =

arg min

p∈4

|S|

kp − fk

(Euclidean projection of

f ∈ R

|S|

onto the

|S|

-dimensional probability simplex

|S|

). Other values of

were studied by Peters et al.

(2019), under the name

α-entmax

transformation. For

α >

1, these transformations have

a propensity for returning sparse distributions, where several entries have zero probability.

Proposition 4 shows that similar properties can be obtained when

is non-ﬁnite (countably

inﬁnite or continuous).

Deformed exponential families.

With a linear parametrization

(

) =

(

), distribu-

tions with the form

(10)

are called deformed exponential families (Naudts, 2009; Sears, 2008),

also referred to as

-exponential families (Ding and Vishwanathan, 2010) and

-exponential

families (Matsuzoe and Ohara, 2012). The geometry of these families induced by Tsallis

entropies was studied by Amari (2016,

4.3).

They include for example

-Student and other

heavy tail distributions (heavy tails arise when

α <

1). Unlike those prior works, in this

paper we are interested in the sparse, light tail scenario (

α >

1), not in heavy tails. For

α >

1, we call these

α-sparse families.

When

α →

-sparse families become exponential

Unfortunately, the literature is inconsistent in deﬁning these coeﬃcients. Our

matches that of Blondel

et al. (2020); Tsallis’

in the context of deformed exponential families equals 2

− α

(which we call

our paper). This family is also related to Amari’s

-divergences, but their

equals 2

q −

1. Inconsistent

deﬁnitions have also been proposed for

-exponential families regarding how they are normalized; for

example, the Tsallis maxent principle leads to a diﬀerent deﬁnition. See Appendix C.4 for details.

Sparse Continuous Distributions and Fenchel-Young Losses

families and they cease to be “sparse”, in the sense that all distributions in the same family

have the same support. Another interesting particular case is that of

= 2, which we will

see in detail in

5. From Proposition 4 and

(7)

, we can see that a 2-sparse family takes the

form (writing p

≡ ˆp

Ω

]):

(t) = [θ

φ(t) − A

(θ) + 1]

, with A

(θ) := A

) = 1 +

−1 +

supp(p

)

φ(t)

|supp(p

. (11)

This generalizes the result of Martins and Astudillo (2016, Proposition 1), who derived a

similar expression for the ﬁnite case.

Gradient of A

A relevant problem is that of characterizing the normalizing function

(

) :=

(

). When

= 1,

(

) =

lim

α→1

(

) =

log

exp

(

)) is the log-

partition function (see

(3)

), and its ﬁrst and higher order derivatives are equal to the

moments of the suﬃcient statistics. The following proposition, stated in Theorem 5 of Amari

and Ohara (2011), and proved in our Appendix C.5, characterizes

(

) for

α 6

= 1 in terms

of an expectation under the β-escort distribution, deﬁned in (8), for β = 2 − α.

Proposition 5 (Gradient of normalizing function A

)

Let

= 2

− α

with

α ∈

2]. Let

˜p

be the

-escort distribution

(8)

. The normalizing function

: Θ

→ R

convex and its gradient coincides with the expectation under the β-escort distribution

∇

(θ) = E

˜p

[φ(t)] =

(t)

φ(t)

(t)

We use this result later in this section to derive the Hessian of Fenchel-Young losses and

in §8 to obtain the Jacobian of entmax attention mechanisms.

To close the loop, we present the following result, proved in Appendix C.6, which relates

Tsallis negentropies Ω

, their convex conjugates Ω

∗

, normalizing functions

(

), and

provides an expression for Ω

-Fenchel-Young losses for linearly parametrized families:

Proposition 6 (Key quantities in α-sparse families)

Let

≡ ˆp

Ω

[

], with

(

) =

(

), and deﬁne as

(

) :=

[

(

)] the “mean parameters” associated with

θ ∈

Θ.

Then the Tsallis negentropy is given by

Ω

) =



µ(θ) − A

(θ)



, (12)

its convex conjugate is given by

Ω

∗

) = (α − 1)Ω

(ˆp

Ω

]) + A

(θ) =



(α − 1)θ

µ(θ) + A

(θ)



, (13)

and the Ω

-Fenchel-Young loss between f

and any p ∈ M

(S) is given by

Ω

, p) = Ω

(p) − Ω

(ˆp

Ω

]) − θ

(v − µ(θ)), (14)

剩余72页未读，继续阅读

易小侠

粉丝: 6550
资源: 9万+

限制域下核心稳定委员会：稀疏连续分布与Fenchel-Young损失研究

IEC61851-23_2014-03

学校共1000名学生,235人住在a宿舍333人在b宿舍432人在c宿舍共有10个宿舍委员按比例层次分配宿舍委员matlan代码

基于语义分割的遥感影像解译开源数据集下载链接

财务 会计相关的论文

Sigrity-Clarity3dWorkbench-USB Connector S-parameter Extraction

前端分析-202307110077

常见算法的 C++11 实现片段，适用于算法竞赛.zip

【SCI2区】龙格库塔优化算法RUN-CNN-GRU-Attention用电需求预测Matlab实现.rar

Kaggle竞赛记录.zip

天池入门竞赛.zip

PRCV2021变化检测竞赛第三名解决方案.zip

开学季相关的主题活动，课程介绍，教程内容，推荐资料.m

基于ssm的大学运动场地管理系统设计与实现.docx

BA_produce_tracker-0.0.3-py3-none-any.whl

Kaggle竞赛实战.zip

数学建模竞赛.zip

遥感图像稀疏表征与智能分析竞赛之语义分割.zip

【SCI2区】黑猩猩优化算法Chimp-CNN-GRU-Attention用电需求预测Matlab实现.rar

BAAlgorithmUtils-1.1.2-py3-none-any.whl

AxonDeepSeg-0.4-py2.py3-none-any.whl.zip

最新资源

财务会计相关的论文