"深度学习课本：Autoencoders原理与实践"

深度学习

需积分: 0 107 浏览量更新于2024-03-22 收藏 24.72MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源推荐

CHAPTER 14. AUTOENCODERS

Figure 14.6:An illustration of the concept of a tangent hyperplane. Here we create a

one-dimensional manifold in 784-dimensional space. We take an MNIST image with 784

pixels and transform it by translating it vertically.The amount of vertical translation

deﬁnes a coordinate along a one-dimensional manifold that traces out a curved path

through image space. This plot shows a few points along this manifold. For visualization,

we have projected the manifold into two dimensional space using PCA. An

-dimensional

manifold has an

-dimensional tangent plane at every point. This tangent plane touches

the manifold exactly at that point and is oriented parallel to the surface at that point.

It deﬁnes the space of directions in which it is possible to move while remaining on

the manifold. This one-dimensional manifold has a single tangent line. We indicate an

example tangent line at one point, with an image showing how this tangent direction

appears in image space. Gray pixels indicate pixels that do not change as we move along

the tangent line, white pixels indicate pixels that brighten, and black pixels indicate pixels

that darken.

520

CHAPTER 14. AUTOENCODERS

0 0.

0 2.

0 4.

0 6.

0 8.

1 0.

r x( )

Identity

Optimal reconstruction

Figure 14.7: If the autoencoder learns a reconstruction function that is invariant to small

perturbations near the data points, it captures the manifold structure of the data. Here

the manifold structure is a collection of -dimensional manifolds. The dashed diagonal0

line indicates the identity function target for reconstruction. The optimal reconstruction

function crosses the identity function wherever there is a data point. The horizontal

arrows at the bottom of the plot indicate the

(

)

− x

reconstruction direction vector

at the base of the arrow, in input space, always pointing towards the nearest “manifold”

(a single datapoint, in the 1-D case). The denoising autoencoder explicitly tries to make

the derivative of the reconstruction function

(

) small around the data points. The

contractive autoencoder does the same for the encoder. Although the derivative of

(

) is

asked to be small around the data points, it can be large between the data points. The

space between the data points corresponds to the region between the manifolds, where

the reconstruction function must have a large derivative in order to map corrupted points

back onto the manifold.

manifold. Such a representation for a particular example is also called its em-

bedding. It is typically given by a low-dimensional vector, with less dimensions

than the “ambient” space of which the manifold is a low-dimensional subset. Some

algorithms (non-parametric manifold learning algorithms, discussed below) directly

learn an embedding for each training example, while others learn a more general

mapping, sometimes called an encoder, or representation function, that maps any

point in the ambient space (the input space) to its embedding.

Manifold learning has mostly focused on unsupervised learning procedures that

attempt to capture these manifolds. Most of the initial machine learning research

on learning nonlinear manifolds has focused on non-parametric methods based on

the nearest-neighbor graph. This graph has one node per training example and

edges connecting near neighbors to each other. These methods (Schölkopf et al.,

1998 Roweis and Saul 2000 Tenenbaum 2000 Brand 2003 Belkin and; , ; et al., ; , ;

521

CHAPTER 14. AUTOENCODERS

14.7 Contractive Autoencoders

The contractive autoencoder ( , , ) introduces an explicit regularizerRifai et al. 2011a b

on the code

(

), encouraging the derivatives of

to be as small as possible:

Ω( ) = h λ



∂f( )x

∂x



. (14.18)

The penalty Ω(

) is the squared Frobenius norm (sum of squared elements) of the

Jacobian matrix of partial derivatives associated with the encoder function.

There is a connection between the denoising autoencoder and the contractive

autoencoder: ( ) showed that in the limit of small GaussianAlain and Bengio 2013

inputnoise,thedenoisingreconstruction errorisequivalent toa contractive

penalty on the reconstruction function that maps

(

)). In other

words, denoising autoencoders make the reconstruction function resist small but

ﬁnite-sized perturbations of the input, while contractive autoencoders make the

feature extraction function resist inﬁnitesimal perturbations of the input. When

using the Jacobian-based contractive penalty to pretrain features

(

) for use

with a classiﬁer, the best classiﬁcation accuracy usually results from applying the

contractive penalty to

(

) rather than to

(

)). A contractive penalty on

(

)

also has close connections to score matching, as discussed in Sec. .14.5.1

The name contractive arises from the way that the CAE warps space. Speciﬁ-

cally, because the CAE is trained to resist perturbations of its input, it is encouraged

to map a neighborhood of input points to a smaller neighborhood of output points.

We can think of this as contracting the input neighborhood to a smaller output

neighborhood.

To clarify, the CAE is contractive only locally—all perturbations of a training

point

are mapped near to

(

). Globally, two diﬀerent points

and

may be

mapped to

(

) and

(

) points that are farther apart than the original points.

It is plausible that

be expanding in-between or far from the data manifolds (see

for example what happens in the 1-D toy example of Fig. ). When the14.7 Ω(

)

penalty is applied to sigmoidal units, one easy way to shrink the Jacobian is to

make the sigmoid units saturate to or . This encourages the CAE to encode0 1

input points with extreme values of the sigmoid that may be interpreted as a

binary code. It also ensures that the CAE will spread its code values throughout

most of the hypercube that its sigmoidal hidden units can span.

We can think of the Jacobian matrix

at a point

as approximating the

nonlinear encoder

(

) as being a linear operator. This allows us to use the word

“contractive” more formally.In the theory of linear operators, a linear operator

524

剩余282页未读，继续阅读

高工-老罗

粉丝: 22
资源: 314

"深度学习课本：Autoencoders原理与实践"

1.deep+learning课本-三巨头(中)1

Deep Learning（深度学习）-3

给我推荐20个比较流行的自动驾驶算法模型代码

deepsort+yolov3-pytorch

为什么使用/deep/ .el-picker-panel修改不了<el-date-picker>组件中的el-picker-panel样式呢？

Research+progress+of+computer-aided+diagnosis+in+cancer+based+on+deep+learning+and+medical+imaging

Deeplearning4j 视频教程链接有哪些

deeplearning4j-core 文档

python train.py --backbone mobilenet --lr 0.007 --workers 1 --epochs 50 --batch-size 8 --gpu-ids 0 --checkname deeplab-mobilenet 是什么意思

:deep(.el-input-group__append, .el-input-group__prepend) { padding: 0; }有什么用

import numpy as np import matplotlib.pyplot as plt plt.style.use('./deeplearning.mplstyle')

/deep/.el-form-item__label

Java 安装deeplearning4j

列举20个ml.net框架下第三方开源项目，关于人脸识别的。并给出下载地址

Java如何使用org.deeplearning4j

deeplearning toolbox 下载方式

[@vue/compiler-sfc] ::v-deep usage as a combinator has been deprecated. Use :deep(<inner-selector>) instead.

/deep/.u-form-item { height: 150px; nvue中如何实现样式穿透

最新资源

:deep(.el-input-groupappend, .el-input-groupprepend) { padding: 0; }有什么用