![](https://csdnimg.cn/release/download_crawler_static/15844069/bg5.jpg)
BOND-TAYLOR ET AL.: ADVANCES IN DEEP GENERATIVE MODELS 5
2.6 Correcting Implicit Generative Models
While EBMs offer powerful representation ability thanks
to unnormalized likelihoods, they can suffer from high
variance training, long training and/or sampling times, and
struggle to support the entire data space. In this section, a
number of hybrid approaches are discussed which address
some of these issues.
2.6.1 Exponential Tilting
To eliminate the need for an EBM to support the entire
space, an EBM can instead be used to correct samples from
an implicit generative network, simplifying the function to
learn and allowing easier sampling. This procedure, referred
to as exponentially tilting an implicit model, is defined as
p
θ,φ
(x) =
1
Z
θ,φ
q
φ
(x)e
−E
θ
(x)
. (14)
By parameterising q
φ
(x) as a latent variable model such as
a normalizing flow [2], [144] or VAE generator [215], MCMC
sampling can be performed in the latent space rather than
the data space. Since the latent space is much simpler, and
often uni-modal, MCMC mixes much more effectively. This
limits the freedom of the model, however, leading some to
jointly sample in latent and data space [2], [215].
2.6.2 Noise Contrastive Estimation
Noise contrastive estimation [47], [64] transforms EBM
training into a classification problem using a noise distri-
bution q
φ
(x) by optimising the loss function,
E
p
d
"
ln
p
θ
(x)
p
θ
(x) + q
φ
(x)
#
+ E
q
φ
"
ln
q
φ
(x)
p
θ
(x) + q
φ
(x)
#
, (15)
where p
θ
(x) = e
E
θ
(x)−c
. This approach can be used to train
a correction via exponential tilting [144], but can also be
used to directly train an EBM and normalizing flow [50].
Equation 15 is equivalent to GAN Equation 24, however,
training formulations differ, with noise contrastive estima-
tion explicitly modelling likelihood ratios.
2.7 Alternative Training Objectives
As aforementioned, energy models trained with contrastive
divergence approximately maximises the likelihood of the
data; likelihood however does not correlate directly with
sample quality [190]. Training EBMs with arbitrary f-
divergences is possible, yielding improved FID scores [220].
While score matching allows EBM training without sam-
pling, score estimates have high variance, an alternative
proposal is to minimise the Stein discrepancy, approximated
by a second network, which requires no sampling and more
closely correlates with likelihood [56]. A middle ground
between denoising score matching and contrastive diver-
gence is diffusion recovery likelihood [10] which can be
optimised via a sequence of denoising EBMs conditioned
on increasingly noisy samples of the data, the conditional
distributions being much easy to MCMC sample from than
typical EBMs [51].
3 VARIATIONAL A UTOENCODERS
One of the key problems associated with energy-based
models is that sampling is not straightforward and mixing
can require a significant amount of time. To circumvent this
issue, it would be beneficial to explicitly sample from the
data distribution with a single network pass.
To this end, suppose we have a latent based model
p
θ
(x|z) with prior p
θ
(z) and posterior p
θ
(z|x); unfortu-
nately optimising this model through max likelihood is in-
tractable due to the integral in p
θ
(x) =
R
z
p
θ
(x|z)p
θ
(z)dz.
Instead, [105] propose learning a second function q
φ
(z|x) =
arg min
q
D
KL
(q
φ
(z|x)||p
θ
(z|x)) which approximates the
true intractable posterior p
θ
(z|x). From the definition of KL
divergence we get
D
KL
(q
φ
(z|x)||p
θ
(z|x)) = E
q
φ
(z|x)
"
ln
q
φ
(z|x)
p
θ
(z|x)
#
= E
q
φ
(z|x)
[ln q
φ
(z|x)] − E
q
φ
(z|x)
[ln p
θ
(z, x)]
+ ln p
θ
(x),
(16)
which can be rearranged to find an alternative definition for
p
θ
(x) that does not require the knowledge of p
θ
(z|x)
ln p
θ
(x) = D
KL
(q
φ
(z|x)||p
θ
(z|x)) − E
q
φ
(z|x)
[ln q
φ
(z|x)]
+ E
q
φ
(z|x)
[ln p
θ
(z, x)]
≥ −E
q
φ
(z|x)
[ln q
φ
(z|x)] + E
q
φ
(z|x)
[ln p
θ
(z, x)]
= −E
q
φ
(z|x)
[ln q
φ
(z|x)] + E
q
φ
(z|x)
[ln p
θ
(z)] (17)
+ E
q
φ
(z|x)
[ln p
θ
(x|z)]
= −D
KL
(q
φ
(z|x)||p
θ
(z)) + E
q
φ
(z|x)
[ln p
θ
(x|z)]
≡ L(θ, φ; x),
where L is known as the evidence lower bound (ELBO) [93].
To optimise this bound with respect to θ and φ, gradients
must be backpropagated through the stochastic process of
generating samples from
˜
z ∼ q
φ
(z|x). This is permitted by
reparameterizing
˜
z using a differentiable function g
φ
(, x)
of a noise variable :
˜
z = g
φ
(, x) with ∼ p() [105].
Monte Carlo gradient estimators can be used to ap-
proximate the expectations, however this yields very
high variance making it impractical. However, if
D
KL
(q
φ
(z|x)||p
θ
(z)) can be integrated analytically then
the variance is manageable. A prior with such a property
needs to be simple enough to sample from but also suffi-
ciently flexible to match the true posterior; a common choice
is a normally distributed prior with diagonal covariance,
z ∼ q
φ
(z|x) = N(z; µ, σ
2
I) with
˜
z = µ + σ and
∼ N(0, I). In this case, the loss simplifies to
˜
L
V AE
(θ, φ; x) '
1
2
J
X
j=1
1 + ln((σ
(j)
)
2
) −(µ
(j)
)
2
−(σ
(j)
)
2
+
1
L
L
X
l=1
ln p
θ
(x|
˜
z
l
). (18)
Despite success on small scale datasets, when applied
to more complex datasets such as natural images, samples
tend to be unrealistic and blurry [40]. This blurriness has
been attributed to the max likelihood objective itself and
MSE reconstruction loss, however, there is evidence that
limited approximation of the true posterior is the root cause