深度学习教程：从 Logistic 回归到卷积神经网络

5星 · 超过95%的资源需积分: 10 14 浏览量更新于2024-07-25 2 收藏 1.29MB PDF 举报

"Lisa Lab的深度学习教程，2013年发布，涵盖了深度学习的基础到进阶内容，包括逻辑回归、多层感知机、卷积神经网络和去噪自编码器等模型的构建与训练。" 这篇深度学习教程详细介绍了深度学习的基本概念和实践方法，适合初学者和有一定基础的学习者。教程主要分为以下几个部分： 1. **LICENSE**：这部分可能包含了教程的使用许可协议，规定了如何可以合法地使用、分发和修改教程内容。 2. **DeepLearningTutorials**：这部分是教程的总览，引导读者进入深度学习的世界，可能包含深度学习的历史、重要性以及基本原理的介绍。 3. **GettingStarted**：这部分提供了入门指导，包括下载教程的步骤、可用的数据集、符号约定、监督学习优化的简介以及使用Theano/Python的提示。这对于初次接触深度学习的人来说是非常实用的。 - **Download**：说明如何获取教程和相关的代码资源。 - **Datasets**：介绍了可用于训练和测试模型的数据集，可能包括MNIST手写数字识别数据集等经典数据集。 - **Notation**：定义了在教程中使用的符号和术语，帮助读者理解后续的数学表述。 - **APrimeronSupervisedOptimizationforDeepLearning**：简述了监督学习中的优化算法，如梯度下降及其变种。 - **Theano/PythonTips**：提供了使用Theano（一个深度学习库）和Python进行编程的建议和技巧。 4. **ClassifyingMNISTdigitsusingLogisticRegression**：这一章通过逻辑回归模型来识别MNIST手写数字，讲解了模型定义、损失函数、模型构建、学习过程和模型测试的完整流程。 5. **MultilayerPerceptron(MLP)**：介绍了多层感知机，包括模型结构、从逻辑回归过渡到MLP、完整的实现步骤以及训练MLP时的技巧。 6. **ConvolutionalNeuralNetworks(LeNet)**：讲解了卷积神经网络（CNN）的原理，包括稀疏连接性、共享权重、细节和符号、卷积操作、最大池化、完整的LeNet模型实现、代码运行以及训练CNN的建议。 7. **DenoisingAutoencoders(dA)**：探讨了自编码器和去噪自编码器，解释了它们的架构和训练过程，以及在数据去噪中的应用。这个教程通过实例和逐步讲解，深入浅出地教授读者如何构建和训练深度学习模型，对于想要了解和掌握深度学习技术的人来说，是一份宝贵的资源。

Deep Learning Tutorial, Release 0.1

L1 and L2 regularization

L1 and L2 regularization involve adding an extra term to the loss function, which penalizes certain parameter

conﬁgurations. Formally, if our loss function is:

NLL(θ, D) = −

|D|

i=0

log P (Y = y

(i)

, θ)

then the regularized loss will be:

E(θ, D) = NLL(θ, D) + λR(θ)

or, in our case

E(θ, D) = NLL(θ, D) + λ||θ||

where

||θ||





|θ|

j=0

|θ





which is the L

norm of θ. λ is a hyper-parameter which controls the relative importance of the regularization

parameter. Commonly used values for p are 1 and 2, hence the L1/L2 nomenclature. If p=2, then the

regularizer is also called “weight decay”.

In principle, adding a regularization term to the loss will encourage smooth network mappings in a neural

network (by penalizing large values of the parameters, which decreases the amount of nonlinearity that

the network models). More intuitively, the two terms (NLL and R(θ)) correspond to modelling the data

well (NLL) and having “simple” or “smooth” solutions (R(θ)). Thus, minimizing the sum of both will, in

theory, correspond to ﬁnding the right trade-off between the ﬁt to the training data and the “generality” of

the solution that is found. To follow Occam’s razor principle, this minimization should ﬁnd us the simplest

solution (as measured by our simplicity criterion) that ﬁts the training data.

Note that the fact that a solution is “simple” does not mean that it will generalize well. Empirically, it

was found that performing such regularization in the context of neural networks helps with generalization,

especially on small datasets. The code block below shows how to compute the loss in python when it

contains both a L1 regularization term weighted by λ

and L2 regularization term weighted by λ

# symbolic Theano variable that represents the L1 regularization term

L1 = T.sum(abs(param))

# symbolic Theano variable that represents the squared L2 term

L2_sqr = T.sum(param

# the loss

loss = NLL + lambda_1

L1 + lambda_2

12 Chapter 3. Getting Started

Deep Learning Tutorial, Release 0.1

Early-Stopping

Early-stopping combats overﬁtting by monitoring the model’s performance on a validation set. A validation

set is a set of examples that we never use for gradient descent, but which is also not a part of the test set. The

validation examples are considered to be representative of future test examples. We can use them during

training because they are not part of the test set. If the model’s performance ceases to improve sufﬁciently

on the validation set, or even degrades with further optimization, then the heuristic implemented here gives

up on much further optimization.

The choice of when to stop is a judgement call and a few heuristics exist, but these tutorials will make use

of a strategy based on a geometrically increasing amount of patience.

# early-stopping parameters

patience = 5000 # look as this many examples regardless

patience_increase = 2 # wait this much longer when a new best is

# found

improvement_threshold = 0.995 # a relative improvement of this much is

# considered significant

validation_frequency = min(n_train_batches, patience/2)

# go through this many

# minibatches before checking the network

# on the validation set; in this case we

# check every epoch

best_params = None

best_validation_loss = numpy.inf

test_score = 0.

start_time = time.clock()

done_looping = False

epoch = 0

while (epoch < n_epochs) and (not done_looping):

# Report "1" for first epoch, "n_epochs" for last epoch

epoch = epoch + 1

for minibatch_index in xrange(n_train_batches):

d_loss_wrt_params = ... # compute gradient

params -= learning_rate

d_loss_wrt_params # gradient descent

# iteration number. We want it to start at 0.

iter = (epoch - 1)

n_train_batches + minibatch_index

# note that if we do ‘iter % validation_frequency‘ it will be

# true for iter = 0 which we do not want. We want it true for

# iter = validation_frequency - 1.

if (iter + 1) % validation_frequency == 0:

this_validation_loss = ... # compute zero-one loss on validation set

if this_validation_loss < best_validation_loss:

# improve patience if loss improvement is good enough

if this_validation_loss < best_validation_loss

improvement_threshold:

3.4. A Primer on Supervised Optimization for Deep Learning 13

Deep Learning Tutorial, Release 0.1

patience = max(patience, iter

patience_increase)

best_params = copy.deepcopy(params)

best_validation_loss = this_validation_loss

if patience <= iter:

done_looping = True

break

# POSTCONDITION:

# best_params refers to the best out-of-sample parameters observed during the optimization

If we run out of batches of training data before running out of patience, then we just go back to the beginning

of the training set and repeat.

Note: The validation_frequency should always be smaller than the patience. The code should

check at least two times how it performs before running out of patience. This is the reason we used the

formulation validation_frequency = min( value, patience/2.)

Note: This algorithm could possibly be improved by using a test of statistical signiﬁcance rather than the

simple comparison, when deciding whether to increase the patience.

3.4.4 Testing

After the loop exits, the best_params variable refers to the best-performing model on the validation set. If

we repeat this procedure for another model class, or even another random initialization, we should use the

same train/valid/test split of the data, and get other best-performing models. If we have to choose what the

best model class or the best initialization was, we compare the best_validation_loss for each model. When

we have ﬁnally chosen the model we think is the best (on validation data), we report that model’s test set

performance. That is the performance we expect on unseen examples.

3.4.5 Recap

That’s it for the optimization section. The technique of early-stopping requires us to partition the set of

examples into three sets (training D

train

, validation D

valid

, test D

test

). The training set is used for minibatch

stochastic gradient descent on the differentiable approximation of the objective function. As we perform

this gradient descent, we periodically consult the validation set to see how our model is doing on the real

objective function (or at least our empirical estimate of it). When we see a good model on the validation set,

we save it. When it has been a long time since seeing a good model, we abandon our search and return the

best parameters found, for evaluation on the test set.

3.5 Theano/Python Tips

3.5.1 Loading and Saving Models

When you’re doing experiments, it can take hours (sometimes days!) for gradient-descent to ﬁnd the best

parameters. You will want to save those weights once you ﬁnd them. You may also want to save your

14 Chapter 3. Getting Started

Deep Learning Tutorial, Release 0.1

current-best estimates as the search progresses.

Pickle the numpy ndarrays from your shared variables

The best way to save/archive your model’s parameters is to use pickle or deepcopy the ndarray objects. So

for example, if your parameters are in shared variables w, v, u, then your save command should look

something like:

>>> import cPickle

>>> save_file = open(’path’, ’wb’) # this will overwrite current contents

>>> cPickle.dump(w.get_value(borrow=True), save_file, -1) # the -1 is for HIGHEST_PROTOCOL

>>> cPickle.dump(v.get_value(borrow=True), save_file, -1) # .. and it triggers much more efficient

>>> cPickle.dump(u.get_value(borrow=True), save_file, -1) # .. storage than numpy’s default

>>> save_file.close()

Then later, you can load your data back like this:

>>> save_file = open(’path’)

>>> w.set_value(cPickle.load(save_file), borrow=True)

>>> v.set_value(cPickle.load(save_file), borrow=True)

>>> u.set_value(cPickle.load(save_file), borrow=True)

This technique is a bit verbose, but it is tried and true. You will be able to load your data and render it in

matplotlib without trouble, years after saving it.

Do not pickle your training or test functions for long-term storage

Theano functions are compatible with Python’s deepcopy and pickle mechanisms, but you should not nec-

essarily pickle a Theano function. If you update your Theano folder and one of the internal changes, then

you may not be able to un-pickle your model. Theano is still in active development, and the internal APIs

are subject to change. So to be on the safe side – do not pickle your entire training or testing functions for

long-term storage. The pickle mechanism is aimed at for short-term storage, such as a temp ﬁle, or a copy

to another machine in a distributed job.

Read more about serialization in Theano, or Python’s pickling.

3.5.2 Plotting Intermediate Results

Visualizations can be very powerful tools for understanding what your model or training algorithm is doing.

You might be tempted to insert matplotlib plotting commands, or PIL image-rendering commands

into your model-training script. However, later you will observe something interesting in one of those pre-

rendered images and want to investigate something that isn’t clear from the pictures. You’ll wished you had

saved the original model.

If you have enough disk space, your training script should save intermediate models and a visualiza-

tion script should process those saved models.

You already have a model-saving function right? Just use it again to save these intermediate models.

Libraries you’ll want to know about: Python Image Library (PIL), matplotlib.

3.5. Theano/Python Tips 15

剩余136页未读，继续阅读

Belial_2010

粉丝: 1335
资源: 46

深度学习教程：从 Logistic 回归到卷积神经网络

Deep Learning Tutorial-李宏毅

Deep Learning Tutorial Release 0.1

Deep Learning Tutorial 李宏毅 一天搞懂深度学习技术原理

deep learning 英文版pdf

用堆叠自编码器对文件夹内的图像进行分类并生成混淆矩阵tensorflow代码

基于cnn的人脸识别python

列出遥感图像目标检测相关文献

全连接神经网络的相关参考文献

可以找一些tensorflow课程吗

用自动化编码器实现低剂量CT重建pytorch代码

最新资源

Deep Learning Tutorial 李宏毅一天搞懂深度学习技术原理