深度学习架构：人工智能的基石

deep

learning

需积分: 0 147 浏览量更新于2024-07-18 收藏 1.08MB PDF 举报

身份认证购VIP最低享 7 折!

30元优惠券

"Learning Deep Architectures for AI" 是一篇由 Yoshua Bengio 撰写的关于深度学习架构的专业论文，探讨了深度学习在人工智能领域的应用和理论优势。文章首先介绍了深度学习架构的基本概念，强调了如何训练这些复杂的模型。作者提出，深度学习的关键在于创建中间表示，即特征和抽象在不同任务之间可以共享，从而提高学习效率和泛化能力。论文还列出了构建人工智能学习系统的一些理想特性，并概述了后续章节的主要内容。在理论优势部分，Bengio 讨论了深度架构的计算复杂性和非正式的推理方法。他指出，尽管深度学习模型的训练通常更为复杂，但其能够处理更高级别的抽象，从而在某些情况下降低计算需求。此外，他还探讨了局部与全局泛化的区别，揭示了模板匹配的局限性，并强调学习分布式表示的重要性。论文详细介绍了神经网络在深度架构中的作用，包括多层神经网络、训练挑战以及无监督学习的应用。特别是在深度生成模型、卷积神经网络（CNN）和自编码器（Auto-Encoder）等方面，Bengio 展示了这些技术如何扩展神经网络的能力，使其能处理更复杂的数据结构和模式。能量模型和玻尔兹曼机（Boltzmann Machines）是深度学习中的另一种重要工具。Bengio 阐述了能量模型和专家产品的关系，以及它们在建模复杂数据分布时的角色。接着，他详细介绍了受限玻尔兹曼机（RBM），这是一种常用的无监督学习模型，通过对比散度等方法进行训练，能够在高维数据中发现有用的特征。 "Learning Deep Architectures for AI" 是一篇深入探讨深度学习理论和技术的权威文献，涵盖了从基础到高级的多个主题，对于理解深度学习在人工智能中的核心地位及其潜在优势有着极大的价值。这篇论文不仅讨论了深度学习的理论背景，还提供了实际模型和算法的详细分析，为读者提供了丰富的学习材料。

资源详情

资源推荐

Theoretical Advantages of Deep Architectures

In this section, we present a motivating argument for the study of

learning algorithms for deep architectures, by way of theoretical results

revealing potential limitations of architectures with insuﬃcient depth.

This part of the monograph (this section and the next) motivates the

algorithms described in the later sections, and can be skipped without

making the remainder diﬃcult to follow.

The main point of this section is that some functions cannot be eﬃ-

ciently represented (in terms of number of tunable elements) by archi-

tectures that are too shallow. These results suggest that it would be

worthwhile to explore learning algorithms for deep architectures, which

might be able to represent some functions otherwise not eﬃciently rep-

resentable. Where simpler and shallower architectures fail to eﬃciently

represent (and hence to learn) a task of interest, we can hope for learn-

ing algorithms that could set the parameters of a deep architecture for

this task.

We say that the expression of a function is compact when it has

few computational elements, i.e., few degrees of freedom that need to

be tuned by learning. So for a ﬁxed number of training examples, and

short of other sources of knowledge injected in the learning algorithm,

14 Theoretical Advantages of Deep Architectures

we would expect that compact representations of the target function

would yield better generalization.

More precisely, functions that can be compactly represented by a

depth k architecture might require an exponential number of computa-

tional elements to be represented by a depth k − 1 architecture. Since

the number of computational elements one can aﬀord depends on the

number of training examples available to tune or select them, the con-

sequences are not only computational but also statistical: poor general-

ization may be expected when using an insuﬃciently deep architecture

for representing some functions.

We consider the case of ﬁxed-dimension inputs, where the computa-

tion performed by the machine can be represented by a directed acyclic

graph where each node performs a computation that is the application

of a function on its inputs, each of which is the output of another node

in the graph or one of the external inputs to the graph. The whole

graph can be viewed as a circuit that computes a function applied to

the external inputs. When the set of functions allowed for the compu-

tation nodes is limited to logic gates, such as {AND, OR, NOT}, this

is a Boolean circuit, or logic circuit.

To formalize the notion of depth of architecture, one must introduce

the notion of a set of computational elements. An example of such a set

is the set of computations that can be performed logic gates. Another

is the set of computations that can be performed by an artiﬁcial neuron

(depending on the values of its synaptic weights). A function can be

expressed by the composition of computational elements from a given

set. It is deﬁned by a graph which formalizes this composition, with

one node per computational element. Depth of architecture refers to

the depth of that graph, i.e., the longest path from an input node to

an output node. When the set of computational elements is the set of

computations an artiﬁcial neuron can perform, depth corresponds to

the number of layers in a neural network. Let us explore the notion of

depth with examples of architectures of diﬀerent depths. Consider the

function f(x)=x ∗ sin(a ∗ x + b). It can be expressed as the composi-

tion of simple operations such as addition, subtraction, multiplication,

The target function is the function that we would like the learner to discover.

2.1 Computational Complexity 17

A two-layer circuit of logic gates can represent any Boolean func-

tion [127]. Any Boolean function can be written as a sum of products

(disjunctive normal form: AND gates on the ﬁrst layer with optional

negation of inputs, and OR gate on the second layer) or a product

of sums (conjunctive normal form: OR gates on the ﬁrst layer with

optional negation of inputs, and AND gate on the second layer). To

understand the limitations of shallow architectures, the ﬁrst result to

consider is that with depth-two logical circuits, most Boolean func-

tions require an exponential (with respect to input size) number of

logic gates [198] to be represented.

More interestingly, there are functions computable with a

polynomial-size logic gates circuit of depth k that require exponential

size when restricted to depth k − 1 [62]. The proof of this theorem

relies on earlier results [208] showing that d-bit parity circuits of depth

2 have exponential size. The d-bit parity function is deﬁned as usual:

parity : (b

,...,b

) ∈{0,1}

→











1, if



i=1

is even

0, otherwise.

One might wonder whether these computational complexity results

for Boolean circuits are relevant to machine learning. See [140] for an

early survey of theoretical results in computational complexity relevant

to learning algorithms. Interestingly, many of the results for Boolean

circuits can be generalized to architectures whose computational ele-

ments are linear threshold units (also known as artiﬁcial neurons [125]),

which compute

f(x)=1



x+b≥0

(2.1)

with parameters w and b. The fan-in of a circuit is the maximum

number of inputs of a particular element. Circuits are often organized

in layers, like multi-layer neural networks, where elements in a layer

only take their input from elements in the previous layer(s), and the

ﬁrst layer is the neural network input. The size of a circuit is the number

of its computational elements (excluding input elements, which do not

perform any computation).

剩余129页未读，继续阅读

totuwei

粉丝: 0
资源: 2

深度学习架构：人工智能的基石

Learning Deep Architectures for AI.pdf

Deep_Learning_Architecture_for_AI.pdf

Deep Learning

Tensorflow_Advance_Techniques:deeplearning.ai Tensorflow先进技术专业化

MIT Deep Learning

deep learning caffe入门

TensorFlow Deep Learning Projects

【Essentials of Deep Learning for Time Series Forecasting】: Tips and Advanced Applications of RNN

Generative Deep Learning.epub

深度学习工具包 DeepLearningToolbox

Machine Learning for Wireless Networks

Machine Learning Projects for Mobile Applications.azw3

Packt.TensorFlow.Deep.Learning.Projects.2018.rar

Hands-On-Deep-Learning-Algorithms-with-Python:Packt的Python动手深度学习算法

Deep Learning Guide】: From Basics to Advanced Applications, Unleashing AI Potential

Fine-Sliced Image Area Segmentation: A Detailed ...from Threshold Segmentation to Deep Learning

: GAN Transfer Learning Applications: Bridging Domain Boundaries to Accelerate AI Innovation

Demystifying Multilayer Perceptrons (MLP): Architecture, Principles, and Applications for Building ...

Cloud-based Machine Learning Model Management: How to Efficiently Supervise Your AI Assets

最新资源