深度学习对计算机架构的影响：一场技术革命

版权申诉

深度学习

5星 · 超过95%的资源 23 浏览量更新于2024-07-19 收藏 2.85MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

深度学习概述-计算机架构》是由Brandon Reagen、Robert Adolf、Paul Whatmough、Gu-Yeon Wei和David Brooks合著的一本专著，收录于Morgan & Claypool出版社的SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE系列第41卷。该书出版于2017年，深入探讨了深度学习在计算机架构领域的应用及其对传统计算机科学的深远影响。深度学习作为一种机器学习方法，特别是针对具有复杂结构的神经网络模型（如深度神经网络），已经彻底改变了诸多计算机科学领域，包括图像识别、语音处理、自然语言理解等。它通过多层非线性变换，解决了诸如分类和回归等历来被认为难以解决的问题，取得了显著的成果。这表明深度学习技术的潜力不仅限于传统的算法优化，而是正在挑战计算机系统的设计与架构。书中详细介绍了深度学习的核心概念，如反向传播算法、卷积神经网络、循环神经网络等，以及这些模型如何在硬件层面实现高效计算。作者们讨论了如何在计算机架构设计中考虑深度学习的需求，比如对于并行计算、内存管理和能耗优化的需求。他们还可能探讨了专用硬件加速器（如GPU和TPU）的重要性，这些硬件能够提供大量的并行处理能力，支持大规模的矩阵运算，这对于深度学习训练的加速至关重要。此外，书中的章节可能涉及如何通过改进芯片设计来支持深度学习，包括但不限于优化数据流、减少延迟、提高能效，以及利用近内存计算、深度学习硬件加速器与传统CPU或GPU的协同工作策略。作者们还可能会讨论深度学习对数据中心和云计算的影响，以及未来的趋势和挑战，比如模型压缩、迁移学习和量子计算的可能性。《深度学习概述-计算机架构》是一本富有洞见的著作，它不仅提供了深度学习的基础理论知识，还深入剖析了这个新兴领域如何与计算机体系结构紧密融合，推动了整个行业的创新和发展。对于计算机架构师、系统工程师以及对深度学习感兴趣的研究人员来说，这本书无疑是一份宝贵的参考资料。

资源详情

资源推荐

10 2. FOUNDATIONS OF DEEP LEARNING

2.1.1 BIOLOGICAL NEURAL NETWORKS

e fundamental unit of the brain is a neuron (Figure 2.2a), a specialized cell that is designed

to pass electrochemical signals to other parts of the nervous system, including other neurons.

A neuron is comprised of three basic pieces: the soma, or body, which contains the nucleus,

metabolic machinery, and other organelles common to all cells; dendrites, which act as a neu-

ron’s inputs, receiving signals from other neurons and from sensory pathways; and an axon, an

elongated extension of the cell responsible for transmitting outgoing signals. Connections be-

tween neurons are formed where an axon of one neuron adjoins the dendrite of another at an

intercellular boundary region called a synapse. e long, spindly structure of neurons enables a

dense web, with some neurons having up to 200,000 diﬀerent direct connections [102].

Axon

Dentrite

Cel

l Body

(Soma)

(a) Annotated visualization of the structure of

a biological neuron, reconstructed from electron

microscope images of 30 nm slices of a mouse

brain [138].

100

-50

0 2 4 6 8 10 12 14

Time (mS)

Membrane Potential (mV)

Relative to Resting State

(b) e shape of an action potential. A small

external voltage stimulus (blue) triggers a cas-

cade of charge buildup inside a neuron (red) via

voltage-gated ion channels. e activation thresh-

old is shown as a dotted line. Simulated using a

Hodgkin-Huxley model of a neuron [73] (Nobel

prize in Medicine, 1963).

Figure 2.2: e structure and operation of biological neurons.

Unlike most biological signaling mechanisms, communication between two neurons is an

electrical phenomenon. e signal itself, called an action potential, is a temporary spike in local

electrical charge diﬀerence across the cell membrane (see Figure 2.2b). is spike is the result

of specialized membrane proteins called voltage-gated channels, which act as voltage-controlled

ion pumps. At rest, these ion channels are closed, but when a small charge diﬀerence builds

up (either through the result of a chemical stimulus from a sensory organ or from an induced

charge from another neuron’s synapse), one type of protein changes shape and locks into an open

position, allowing sodium ions to ﬂow into the cell, amplifying the voltage gradient. When the

voltage reaches a certain threshold, the protein changes conformation again, locking itself into a

closed state. Simultaneously, a second transport protein switches itself open, allowing potassium

2.1. NEURAL NETWORKS 11

ions to ﬂow out of the cell, reversing the voltage gradient until it returns to its resting state. e

locking behavior of voltage-gated channels causes hysteresis: once a spike has started, any further

external stimuli will be dominated by the rush of ions until the entire process has returned to its

rest state. is phenomenon, combined with the large local charge gradient caused by the ion

ﬂow, allows an action potential to propagate unidirectionally along the surface of a neuron. is

forms the fundamental mechanism of communication between neurons: a small signal received

at dendrites translates into a wave of charge ﬂowing through the cell body and down the axon,

which in turn causes a buildup of charge at the synapses of other neurons to which it is connected.

e discriminative capacity of a biological neuron stems from the mixing of incoming

signals. Synapses actually come in two types: excitatory and inhibitory, which cause the outgoing

action potential sent from one neuron to induce either an increase or decrease of charge on the

dendrite of an adjoining neuron. ese small changes in charge accumulate and decay over short

time scales until the overall cross-membrane voltage reaches the activation threshold for the ion

channels, triggering an action potential. At a system level, these mutually induced pulses of

electrical charge combine to create the storm of neural activity that we call thought.

2.1.2 ARTIFICIAL NEURAL NETWORKS

Computational and mathematical models of neurons have a long, storied history, but most ef-

forts can be broadly grouped into two categories: (1) models that replicate biological neurons to

explain or understand their behavior or (2) solve arbitrary problems using neuron-inspired mod-

els of computation. e former is typically the domain of biologists and cognitive scientists, and

computational models that fall into this group are often described as neuromorphic computing, as

the primary goal is to remain faithful to the original mechanisms. is book deals exclusively

with the latter category, in which loosely bio-inspired mathematical models are brought to bear

on a wide variety of unrelated, everyday problems. e two ﬁelds do share a fair amount of

common ground, and both are important areas of research, but the recent rise in practical appli-

cation of neural networks has been driven largely by this second area. ese techniques do not

solve learning problems in the same way that a human brain does, but, in exchange, they can

oﬀer other advantages, such as being simpler for a human to build or mapping more naturally

to modern microprocessors. For brevity, we use “neural network” to refer to this second class of

algorithms in the rest of this book.

We’ll start by looking at a single artiﬁcial neuron. One of the earliest and still most widely

used models is the perceptron, ﬁrst proposed by Rosenblatt in 1957. In modern language, a per-

ceptron is just two terms: a weighted sum of inputs x

, and a nonlinear activation function, ':

y D '

e vestiges of a biological neuron are visible: the summation reﬂects charge accumu-

lation in the soma, and ' loosely models the activation threshold of voltage-gated membrane

12 2. FOUNDATIONS OF DEEP LEARNING

proteins. Rosenblatt, himself a psychologist, developed perceptrons as a form of neuromorphic

computing, as a way of formally describing how simple elements collectively produce higher-

level behavior [118]. However, individual perceptrons can just as easily be viewed as generalized

linear classiﬁers when ' has discrete outputs:

is a linear combination of inputs and '

is a mapping function. For instance, the linear classiﬁer in Figure 2.3a is a weighted sum of the

x- and y-coordinates of the points, and ' is just a comparison to a ﬁxed threshold (sums above

the threshold are red; those below, blue). In point of fact, many diﬀerent interpretations of this

equation exist, largely because it is so simple. is highlights a fundamental principle of neu-

ral networks: the ability of a neural net to model complex behavior is not due to sophisticated

neurons, but to the aggregate behavior of many simple parts.

(a) Points in R

, subdivided by a single linear

classiﬁer. One simple way of understanding lin-

ear classiﬁers is as a line (or hyperplane, in higher

dimensions) that splits space into two regions. In

this example, points above the line are mapped to

class 1 (red); those below, to class 0 (blue).

(b) Points in R

, subdivided by a combination

of four linear classiﬁers. Each classiﬁer maps all

points to class 0 or 1, and an additional linear clas-

siﬁer is used to combine the four. is hierarchical

model is strictly more expressive than any linear

classiﬁer by itself.

Figure 2.3: Simple elements can be combined to express more complex relationships. is is one

basic tenet of deep neural networks.

e simplest neural network is called a multilayer perceptron or MLP. e structure is what

it sounds like: we organize many parallel neurons into a layer and stack multiple layers together

in sequence. Figure 2.4 shows two equivalent visual representations. e graph representation

emphasizes connectivity and data dependencies. Each neuron is represented as a node, and each

weight is an edge. Neurons on the left are used as inputs to neurons on their right, and values

“ﬂow” to the right. Alternately, we can focus on the values involved in the computation. Ar-

ranging the weights and input/outputs as matrices and vectors, respectively, leads us to a matrix

representation. When talking about neural networks, practitioners typically consider a “layer”

to encompass the weights on incoming edges as well as the activation function and its output.

is can sometimes cause confusion to newcomers, as the inputs to a neural network are often

referred to as the “input layer”, but it is not counted toward the overall depth (number of lay-

14 2. FOUNDATIONS OF DEEP LEARNING

a two-layer MLP behave? We start with the expressions for both layers, using '.x/ D x:

D w

Now we simply substitute

D w

Since matrix multiplication is associative and w

are constant, we can reduce the product to a

new matrix

D w

is leaves us with an aggregate expression,which has the same form as each original layer. is is

a well-known identity: composing any number of linear transformations produces another linear

transformation. It implies that without a nonlinear activation function ', a neural network of

any depth is identical to some single-layer network. In other words, without ', depth is a just

a pretense of complexity. What might be surprising is that even simple nonlinear functions are

suﬃcient to enable complex behavior. e deﬁnition of the widely used ReLU function is just

the positive component of its input:

ReLU

.x/ D

(

x  0 W 0

x > 0 W x

In practice, neural networks can become much more complicated than the multilayer per-

ceptrons introduced here, and often they contain a variety of types of layers (see Chapter 3). Fun-

damentally, though, the underlying premise remains the same: complexity is achieved through

a combination of simpler elements broken up by nonlinearities.

2.1.3 DEEP NEURAL NETWORKS

As a linear classiﬁer, a single perceptron is actually a fairly primitive model, and it is fairly easy

to come up with noncontrived examples for which a perceptron fairs poorly, the most famous

being boolean XOR [105]. Historically, this was a source of contention in the AI community

in the late 1960s, and it was in part responsible for casting doubt on the viability of artiﬁcial

neural networks for most of the 1970s. us, it was a surprise in 1989 when several indepen-

dent researchers discovered that even a single hidden layer is provably suﬃcient for an MLP to

express any continuous function with arbitrary precision [37, 49, 74]. Unfortunately, this result,

called the Universal Approximation eorem, says nothing about how to construct or tune such

a network, nor how eﬃcient it will be in computing it. As it turns out, expressing complicated

剩余107页未读，继续阅读

Fun_He

粉丝: 18
资源: 104

深度学习对计算机架构的影响：一场技术革命

Deep Learning for Computer Architects

神经计算机体系结构

d2l-zh-pytorch-2.0.0.pdf

cudnn-11.2-windows-x64-v8.1.1.33.zip

opencv-contrib-python3.4.1.15

cudnn-10.0-linux-x64-v7.6.5.32.tgz 下载

xception-b5690688.pth

cudnn-10.0-windows10-x64-v7.6.5.32.zip

深度学习题库大全-hr.docx

给我一些人工智能书籍的推荐

cudnn-10.2-windows10-x64-v7.6.5.32.zip

xposed-v89-sdk25-arm64.zip下载

envi deep learning v1.0深度学习操作教程_202003.pdf

yolov9-c-converted.pt 和 yolov9-c.pt 有什么区别

deep-learning-with-pytorch.pdf 15章

斯坦福大学-深度学习基础教程.pdf

pt_inception-2015-12-05-6726825d.pthxiazailujing

人工智能会用到的常见英文以及对应的中文

deeplearning深度学习笔记v5.72.pdf

cudnn-10.1-windows10-x64-v7.6.5.32.zip

最新资源