C++与CUDA C实现深度学习：卷1 - RBMs与监督网络

需积分: 10 137 浏览量更新于2024-07-18 收藏 3.96MB PDF 举报

"Deep Belief Nets in C++ and CUDA C Volume 1--2018" 是一本关于深度学习的书籍，由 Timothy Masters 撰写，主要讲解如何使用 C++ 和 CUDA C 实现深度信念网络（Deep Belief Nets）。该书涵盖了受限玻尔兹曼机（Restricted Boltzmann Machines, RBMs）和监督前馈网络的实现，并强调了这些技术的重要性。深度信念网络（Deep Belief Nets, DBNs）是一种多层的生成模型，由多个受限玻尔兹曼机（Restricted Boltzmann Machines, RBMs）堆叠而成。在本书中，作者将详细介绍如何使用 C++ 和 CUDA C 这两种编程语言来构建和训练这些网络。CUDA C 是 NVIDIA 提供的一种用于加速 GPU 计算的编程接口，它使得在大规模数据处理和深度学习中能够利用 GPU 的并行计算能力。 1. **受限玻尔兹曼机 (Restricted Boltzmann Machines)**：RBM 是一种二态马尔可夫随机场，它包含两层神经元：可见层和隐藏层。每个神经元可以处于活动或非活动状态。RBM 可以用于特征学习，通过无监督学习方法进行预训练，然后作为初始化权重用于监督学习的深度网络。在本书中，读者将学习如何在 C++ 和 CUDA C 中实现 RBMs 的训练和反向传播算法。 2. **监督前馈网络 (Supervised Feedforward Networks)**：这是最常见的深度学习模型之一，包括输入层、隐藏层和输出层。在监督学习中，网络通过调整权重来最小化预测输出与实际标签之间的误差。作者会讲解如何构建和训练这种网络，以及如何将其与 RBMs 结合使用，形成深度信念网络。 3. **生成采样 (Generative Sampling)**：在 RBMs 中，生成采样是通过从隐藏层到可见层的正向传播和从可见层到隐藏层的反向传播过程来生成新的数据样本。这一过程在书中会被详细阐述，它对于理解 RBMs 如何学习数据的潜在表示至关重要。 4. **CUDA C 应用**：CUDA C 的使用使得在 GPU 上并行执行深度学习算法成为可能，大大提高了计算效率。读者将学习如何编写高效的 CUDA 内核，以及如何管理 GPU 内存，以优化深度学习模型的训练速度。 5. **实践应用**：除了理论知识，本书还可能包含实际案例和练习，帮助读者将所学应用于解决实际问题，如图像分类、语音识别或自然语言处理等。通过学习本书，读者不仅可以掌握深度学习的基础理论，还将获得使用 C++ 和 CUDA C 进行高效深度学习实现的实践经验，这对于在硬件资源有限或者需要高性能计算的场景下进行深度学习研究和开发非常有价值。

2) For each training case, use the activations of the trained hidden

layer as inputs to another hidden layer. Teach this second hidden

layer to reectively reproduce its inputs.

3) Repeat this for as many hidden layers as desired. Each new hidden

layer encodes higher-level patterns. For example, the raw data

input may be a rectangular array of gray-level pixels. e rst

hidden layer may detect edges in various parts of the image. e

second hidden layer may detect patterns of connection of these

edges. A third hidden layer may assemble recognizable objects

out of the connected edges. Such progressive feature discovery is

impossible in a shallow network.

4) Use supervised training to map the activations of the hidden

neurons in the last (topmost) hidden layer to classes or

predictions. Alternatively, use the weights learned in the

unsupervised training as initial values for supervised training of a

traditional but very deep feedforward neural network.

Many variations of this general algorithm are possible, and we will explore several of

the most popular and useful variations in this book. But in a broad sense, the algorithm

described here covers much of the application of deep belief nets today.

One of the most fascinating properties of deep belief nets is their remarkable ability

to generalize beyond the universe of training examples. This is likely because the

output layer, rather than seeing the raw data, is seeing “universal” patterns in the raw

data—patterns that because of their universality are likely to reappear in the general

population.

A closely related property is that deep belief nets are shockingly robust against

overfitting. Every beginning statistics student learns the importance of using many, many

more training cases than optimizable parameters. The standard wisdom is that if one

uses 100 cases to train a model with 50 optimizable parameters, the resulting model will

learn as much about the noise in the training set as it learns about the legitimate patterns

and will hence be worthless. But consider the MNIST dataset of handwritten digits. Its

standard training set has 60,000 training cases and 10,000 independent test cases. One

can use these 60,000 cases to train a deep belief net that has several million optimizable

parameters and yet achieve on the order of 1 percent error rate on the test set!

Chapter 1 IntroduCtIon

Suppose we have K output neurons numbered 1 through K. For a given training case,

let t

be the true value for this case, the value that we hope the network will produce, and

let o

be the output actually obtained. Then the mean squared error (MSE) for this single

case is given by Equation 2-1. To compute the MSE for the entire training set, sum this

quantity for all cases and divide by the number of cases.

()

(2-1)

Supervised training of a multiple-layer feedforward network amounts to finding

the weights and bias terms that minimize Equation 2-1. In any numerical minimization

algorithm, it is of great benefit to be able to efficiently compute the gradient, the partial

derivatives of the criterion being minimized with respect to each individual parameter.

Luckily, this is quite easy in this application. We just start at the output layer and work

backward, repeatedly invoking the chain rule of differentiation.

The activation of output neuron k is just a weighted sum of the activations of the

neurons in the prior layer. We’ll use the Greek letter delta to designate the derivative of

the error with respect to this weighted sum coming into output neuron k. Examination

of Equation 2-1 shows this to simply be as shown in Equation 2-2. In this equation, the

superscript O means that this signifies the delta of an output neuron.

ot=-

()

(2-2)

This neuron is receiving a weighted sum of activations from all neurons in the

prior layer. How can we compute the derivative of the error with respect to the weight

from neuron i in the prior layer? The simple chain rule says that this is the product of

the derivative in Equation 2-2 times the derivative of the net input (the weighted sum

coming into this output neuron) with respect to this weight.

This latter term is trivial. The contribution to the weighted sum from neuron i in

the prior layer is just the activation of that neuron times the weight connecting it to the

output neuron k. We shall designate this output weight as w

. So, the derivative of that

weighted sum with respect to w

is just the activation of neuron i. This leads us to the

formula for the partial derivatives of the error with respect to the weights connecting the

last hidden layer to the output layer. In Equation 2-3 we use the superscript M on a to

Chapter 2 SuperviSed FeedForward NetworkS

indicate that it is the activation of a neuron in hidden-layer M, where there are M hidden

layers numbered from 1 through M.

¶¶

(2-3)

There are two complications when we deal with the weights feeding hidden layers.

Let’s consider the weights leading from hidden-layer M−1 to hidden-layer M, the last

hidden layer. We ultimately want the partial derivatives of the error with respect to each

of these weights. As when dealing with the output layer, we’ll split this derivative into the

product of the derivative of the net input feeding this neuron with respect to the weight

times the derivative of the error with respect to this neuron’s net input. As before, the

former term here is trivial; it’s just the activation of the prior neuron feeding through this

weight. It’s the latter that’s messy.

The first complication is that the outputs are linear: the output activation is just the

net input to that output neuron. But the hidden neurons are nonlinear. In particular,

the function that maps the net input of a hidden neuron to its activation is the logistic

function shown in Equation 1-4. So, the chain rule states that the derivative of the error

with respect to the net input is the derivative of the error with respect to the output

times the derivative of the output with respect to the input. Luckily, the derivative of the

logistic function f (a) is simple, as shown in Equation 2-4.

()

fa fa fa1

(2-4)

The remaining term is more complicated because the output of a neuron in a hidden

layer feeds into every neuron in the next layer and thus impacts the error through

every one of those paths. Recall that δ

is the derivative of the error with respect to

the weighted sum coming into output neuron k. The contribution to this weighted

sum going into output neuron k from neuron i in the prior layer M is the activation of

hidden neuron i times the weight connecting it to output neuron k. So, the impact on

the derivative of the error from the activation of neuron i that goes through this path is

times the connecting weight. Since neuron i impacts the error through all output

neurons, we must sum these contributions, as shown in Equation 2-5.

¶¶

(2-5)

Chapter 2 SuperviSed FeedForward NetworkS

www.allitebooks.com

剩余224页未读，继续阅读

白鸽-小蛋

粉丝: 10
资源: 166

C++与CUDA C实现深度学习：卷1 - RBMs与监督网络

Deep Belief Nets in C++ and CUDA C: Volume 3: Convolutional Nets

Deep Belief Nets in C++ and CUDA C Volume 2 Autoencoding in the Complex epub

Deep Belief Nets in C++ and CUDA C Volume 1 Restricted Boltzmann 无水印原版pdf

Deep Belief Nets in C++ and CUDA C_Volume 1-Apress(2018)

Deep Belief Nets in C++ and CUDA C Volume

Deep Belief Nets in C++ and CUDA C Volume 1 Restricted Boltzmann Machines epub

Deep Belief Nets in C++ and CUDA C Volume 2 Autoencoding in the Complex 无水印原版pdf

Deep Belief Nets in C and CUDA C Volume 1.pdf

深度信念网络C++与CUDA实现：卷1

Data Mining Algorithms in C++

最新资源