深度网络的贪心分层训练策略
需积分: 9 63 浏览量
更新于2024-09-09
收藏 310KB PDF 举报
"这篇资源主要讨论了贪婪分层训练（Greedy Layer-Wise Training）在深度网络中的应用，特别是如何解决深度神经网络的训练难题。文章由Yoshua Bengio等人撰写，他们来自蒙特利尔大学，研究关注于如何通过逐层无监督学习策略改进深度信念网络（Deep Belief Networks, DBN）的训练效果。
深度网络由于其多层非线性结构，理论上可以更高效地表示某些复杂函数，有时甚至在计算元素需求上比浅层架构有指数级优势。然而，传统的基于梯度优化的方法在随机初始化时往往陷入局部最优解，导致训练困难。Hinton等人提出的贪婪分层无监督学习算法为DBN的训练提供了新思路。DBN是一种具有多层隐藏因果变量的生成模型。
作者们对这一算法进行了实证研究，探讨了其成功的原因，并尝试扩展到输入是连续数据或输入分布结构不明确的监督任务中。实验结果证实，贪婪分层无监督训练策略主要通过将权重初始化在接近良好局部最小值的区域，从而生成内部的分布式表示，这些表示是输入的高级抽象，有助于提高泛化能力。
文章进一步探索了算法的变体，旨在更好地理解其工作原理并扩大其适用范围。这表明，这种分层训练方法对于优化过程非常有益，能帮助深度网络在初始阶段就建立有利的权重配置，进而提升模型的整体性能。"
该资源主要涵盖了以下几个知识点：
1. 深度网络的优势：在表示复杂函数时，深度多层神经网络的效率可能远高于浅层网络。
2. 训练深度网络的挑战：梯度下降法在随机初始化时可能陷入局部最优，难以收敛至全局最优。
3. 贪婪分层训练：Hinton等人的解决方案，通过逐层无监督预训练初始化权重。
4. 深度信念网络（DBN）：一种多层隐藏单元的生成模型，适用于贪婪分层训练。
5. 实验与扩展：算法的实证研究，包括对连续输入和输入分布不明确情况的适应性。
6. 分布式表示与泛化：贪婪分层训练产生的内部表示有助于提高模型的泛化能力。
7. 算法变体：对原始算法的改进和扩展，以增进理解和应用范围。
这篇资源对于理解和改进深度学习模型的训练策略，尤其是深度信念网络的训练，具有重要的理论和实践价值。
The layer-to-layer conditionals associated with the RBM factorize like in (1) and give rise to
P (v
k
= 1|h) = sigm(b
k
+
P
j
W
jk
h
j
) and Q(h
j
= 1|v) = sigm(c
j
+
P
k
W
jk
v
k
).
2.2 Gibbs Markov chain and log-likelihood gradient in an RBM
To obtain an estimator of the gradient on the log-likelihood of an RBM, we consider a Gibbs Markov
chain on the (visible units, hidden units) pair of variables. Gibbs sampling from an RBM proceeds by
sampling h given v, then v given h, etc. Denote v
t
for the t-th v sample from that chain, starting at
t = 0 with v
0
, the “input observation” for the RBM. Therefore, (v
k
, h
k
) for k → ∞ is a sample from
the joint P (v, h). The log-likelihood of a value v
0
under the model of the RBM is
log P (v
0
) = log
X
h
P (v
0
, h) = log
X
h
e
−energy(v
0
,h)
− log
X
v,h
e
−energy(v,h)
and its gradient with respect to θ = (W, b, c) is
∂ log P (v
0
)
∂θ
= −
X
h
0
Q(h
0
|v
0
)
∂energy(v
0
, h
0
)
∂θ
+
X
v
k
,h
k
P (v
k
, h
k
)
∂energy(v
k
, h
k
)
∂θ
for k → ∞. An unbiased sample is −
∂energy(v
0
, h
0
)
∂θ
+ E
h
k
∂energy(v
k
, h
k
)
∂θ
|v
k
,
where h
0
is a sample from Q(h
0
|v
0
) and (v
k
, h
k
) is a sample of the Markov chain, and the expecta-
tion can be easily computed thanks to P (h
k
|v
k
) factorizing. The idea of the Contrastive Divergence
algorithm (Hinton, 2002) is to take k small (typically k = 1). A pseudo-code for Contrastive Di-
vergence training (with k = 1) of an RBM with binomial input and hidden units is presented in the
Appendix (Algorithm RBMupdate(x, , W, b, c)). This procedure is called repeatedly with v
0
= x
sampled from the training distribution for the RBM. To decide when to stop one may use a proxy for
the training criterion, such as the reconstruction error − log P (v
1
= x|v
0
= x).
2.3 Greedy layer-wise training of a DBN
A greedy layer-wise training algorithm was proposed (Hinton et al., 2006) to train a DBN one layer at
a time. One first trains an RBM that takes the empirical data as input and models it. Denote Q(g
1
|g
0
)
the posterior over g
1
associated with that trained RBM (we recall that g
0
= x with x the observed
input). This gives rise to an “empirical” distribution bp
1
over the first layer g
1
, when g
0
is sampled
from the data empirical distribution bp: we have bp
1
(g
1
) =
X
g
0
bp(g
0
)Q(g
1
|g
0
).
Note that a 1-level DBN is an RBM. The basic idea of the greedy layer-wise strategy is that after
training the top-level RBM of a `-level DBN, one changes the interpretation of the RBM parameters
to insert them in a (` + 1)-level DBN: the distribution P (g
`−1
|g
`
) from the RBM associated with
layers ` − 1 and ` is kept as part of the DBN generative model. In the RBM between layers ` − 1
and `, P (g
`
) is defined in terms on the parameters of that RBM, whereas in the DBN P(g
`
) is defined
in terms of the parameters of the upper layers. Consequently, Q(g
`
|g
`−1
) of the RBM does not
correspond to P (g
`
|g
`−1
) in the DBN, except when that RBM is the top layer of the DBN. However,
we use Q(g
`
|g
`−1
) of the RBM as an approximation of the posterior P (g
`
|g
`−1
) for the DBN.
The samples of g
`−1
, with empirical distribution bp
`−1
, are converted stochastically into samples of g
`
with distribution bp
`
through bp
`
(g
`
) =
P
g
`−1
bp
`−1
(g
`−1
)Q(g
`
|g
`−1
). Although bp
`
cannot be rep-
resented explicitly it is easy to sample unbiasedly from it: pick a training example and propagate it
stochastically through the Q(g
i
|g
i−1
) at each level. As a nice side benefit, one obtains an approxi-
mation of the posterior for all the hidden variables in the DBN, at all levels, given an input g
0
= x.
Mean-field propagation (see below) gives a fast deterministic approximation of posteriors P (g
`
|x).
Note that if we consider all the layers of a DBN from level i to the top, we have a smaller DBN,
which generates the marginal distribution P (g
i
) for the complete DBN. The motivation for the greedy
procedure is that a partial DBN with ` − i levels starting above level i may provide a better model for
P (g
i
) than does the RBM initially associated with level i itself.
The above greedy procedure is justified using a variational bound (Hinton et al., 2006). As a con-
sequence of that bound, when inserting an additional layer, if it is initialized appropriately and has
enough units, one can guarantee that initial improvements on the training criterion for the next layer
剩余12页未读，继续阅读
2014-08-22 上传
2024-01-08 上传
点击了解资源详情
2024-11-07 上传
2024-11-07 上传
2024-11-07 上传
2024-11-07 上传
2024-11-07 上传
MilleLee
- 粉丝: 0
- 资源: 1
我的内容管理
展开
最新资源
- 探索数据转换实验平台在设备装置中的应用
- 使用git-log-to-tikz.py将Git日志转换为TIKZ图形
- 小栗子源码2.9.3版本发布
- 使用Tinder-Hack-Client实现Tinder API交互
- Android Studio新模板：个性化Material Design导航抽屉
- React API分页模块：数据获取与页面管理
- C语言实现顺序表的动态分配方法
- 光催化分解水产氢固溶体催化剂制备技术揭秘
- VS2013环境下tinyxml库的32位与64位编译指南
- 网易云歌词情感分析系统实现与架构
- React应用展示GitHub用户详细信息及项目分析
- LayUI2.1.6帮助文档API功能详解
- 全栈开发实现的chatgpt应用可打包小程序/H5/App
- C++实现顺序表的动态内存分配技术
- Java制作水果格斗游戏：策略与随机性的结合
- 基于若依框架的后台管理系统开发实例解析
