BOND-TAYLOR ET AL.: ADVANCES IN DEEP GENERATIVE MODELS 5
2.6 Correcting Implicit Generative Models
While EBMs offer powerful representation ability thanks
to unnormalized likelihoods, they can suffer from high
variance training, long training and/or sampling times, and
struggle to support the entire data space. In this section, a
number of hybrid approaches are discussed which address
some of these issues.
2.6.1 Exponential Tilting
To eliminate the need for an EBM to support the entire
space, an EBM can instead be used to correct samples from
an implicit generative network, simplifying the function to
learn and allowing easier sampling. This procedure, referred
to as exponentially tilting an implicit model, is defined as
p
θ,φ
(x) =
1
Z
θ,φ
q
φ
(x)e
−E
θ
(x)
. (14)
By parameterising q
φ
(x) as a latent variable model such as
a normalizing flow [2], [144] or VAE generator [215], MCMC
sampling can be performed in the latent space rather than
the data space. Since the latent space is much simpler, and
often uni-modal, MCMC mixes much more effectively. This
limits the freedom of the model, however, leading some to
jointly sample in latent and data space [2], [215].
2.6.2 Noise Contrastive Estimation
Noise contrastive estimation [47], [64] transforms EBM
training into a classification problem using a noise distri-
bution q
φ
(x) by optimising the loss function,
E
p
d
"
ln
p
θ
(x)
p
θ
(x) + q
φ
(x)
#
+ E
q
φ
"
ln
q
φ
(x)
p
θ
(x) + q
φ
(x)
#
, (15)
where p
θ
(x) = e
E
θ
(x)−c
. This approach can be used to train
a correction via exponential tilting [144], but can also be
used to directly train an EBM and normalizing flow [50].
Equation 15 is equivalent to GAN Equation 24, however,
training formulations differ, with noise contrastive estima-
tion explicitly modelling likelihood ratios.
2.7 Alternative Training Objectives
As aforementioned, energy models trained with contrastive
divergence approximately maximises the likelihood of the
data; likelihood however does not correlate directly with
sample quality [190]. Training EBMs with arbitrary f-
divergences is possible, yielding improved FID scores [220].
While score matching allows EBM training without sam-
pling, score estimates have high variance, an alternative
proposal is to minimise the Stein discrepancy, approximated
by a second network, which requires no sampling and more
closely correlates with likelihood [56]. A middle ground
between denoising score matching and contrastive diver-
gence is diffusion recovery likelihood [10] which can be
optimised via a sequence of denoising EBMs conditioned
on increasingly noisy samples of the data, the conditional
distributions being much easy to MCMC sample from than
typical EBMs [51].
3 VARIATIONAL A UTOENCODERS
One of the key problems associated with energy-based
models is that sampling is not straightforward and mixing
can require a significant amount of time. To circumvent this
issue, it would be beneficial to explicitly sample from the
data distribution with a single network pass.
To this end, suppose we have a latent based model
p
θ
(x|z) with prior p
θ
(z) and posterior p
θ
(z|x); unfortu-
nately optimising this model through max likelihood is in-
tractable due to the integral in p
θ
(x) =
R
z
p
θ
(x|z)p
θ
(z)dz.
Instead, [105] propose learning a second function q
φ
(z|x) =
arg min
q
D
KL
(q
φ
(z|x)||p
θ
(z|x)) which approximates the
true intractable posterior p
θ
(z|x). From the definition of KL
divergence we get
D
KL
(q
φ
(z|x)||p
θ
(z|x)) = E
q
φ
(z|x)
"
ln
q
φ
(z|x)
p
θ
(z|x)
#
= E
q
φ
(z|x)
[ln q
φ
(z|x)] − E
q
φ
(z|x)
[ln p
θ
(z, x)]
+ ln p
θ
(x),
(16)
which can be rearranged to find an alternative definition for
p
θ
(x) that does not require the knowledge of p
θ
(z|x)
ln p
θ
(x) = D
KL
(q
φ
(z|x)||p
θ
(z|x)) − E
q
φ
(z|x)
[ln q
φ
(z|x)]
+ E
q
φ
(z|x)
[ln p
θ
(z, x)]
≥ −E
q
φ
(z|x)
[ln q
φ
(z|x)] + E
q
φ
(z|x)
[ln p
θ
(z, x)]
= −E
q
φ
(z|x)
[ln q
φ
(z|x)] + E
q
φ
(z|x)
[ln p
θ
(z)] (17)
+ E
q
φ
(z|x)
[ln p
θ
(x|z)]
= −D
KL
(q
φ
(z|x)||p
θ
(z)) + E
q
φ
(z|x)
[ln p
θ
(x|z)]
≡ L(θ, φ; x),
where L is known as the evidence lower bound (ELBO) [93].
To optimise this bound with respect to θ and φ, gradients
must be backpropagated through the stochastic process of
generating samples from
˜
z ∼ q
φ
(z|x). This is permitted by
reparameterizing
˜
z using a differentiable function g
φ
(, x)
of a noise variable :
˜
z = g
φ
(, x) with ∼ p() [105].
Monte Carlo gradient estimators can be used to ap-
proximate the expectations, however this yields very
high variance making it impractical. However, if
D
KL
(q
φ
(z|x)||p
θ
(z)) can be integrated analytically then
the variance is manageable. A prior with such a property
needs to be simple enough to sample from but also suffi-
ciently flexible to match the true posterior; a common choice
is a normally distributed prior with diagonal covariance,
z ∼ q
φ
(z|x) = N(z; µ, σ
2
I) with
˜
z = µ + σ and
∼ N(0, I). In this case, the loss simplifies to
˜
L
V AE
(θ, φ; x) '
1
2
J
X
j=1
1 + ln((σ
(j)
)
2
) −(µ
(j)
)
2
−(σ
(j)
)
2
+
1
L
L
X
l=1
ln p
θ
(x|
˜
z
l
). (18)
Despite success on small scale datasets, when applied
to more complex datasets such as natural images, samples
tend to be unrealistic and blurry [40]. This blurriness has
been attributed to the max likelihood objective itself and
MSE reconstruction loss, however, there is evidence that
limited approximation of the true posterior is the root cause
剩余20页未读，继续阅读
syp_net
- 粉丝: 158
- 资源: 1187
我的内容管理
展开
最新资源
- Vue实现iOS原生Picker组件：详细解析与实现思路
- Arduino蓝牙小车：参数调试与功能控制
- 百度Java面试精华：200页精选资源涵盖核心知识点
- Swift使用CoreData填坑指南：CoreData在Swift 3.0的变化
- 微距离无线充电器创新设计及其实验探索
- MTK Android平台开发全攻略：44步详解流程
- RecyclerView全面解析：替代ListView的新选择
- Android开发：自动适配中英文键盘解决方案
- Android调用WebService接口教程
- Android开发：BitmapUtil图片处理全解析与实例
- Android多线程断点续传实现详解
- PCA算法在人脸识别会议签到系统中的应用
- EventBus 3.0：Android事件总线详解与实战应用
- Android FileUtil：全面解析文件操作实用技巧与实例
- RecyclerView添加头部和尾部实战教程
- Android实现微博滑动固定顶部栏实战与优化
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
