深度学习入门：从逻辑计算到人工智能

需积分: 10 128 浏览量更新于2024-07-18 1 收藏 3.71MB PDF 举报

"《Introduction to Deep Learning 2017》是Sandro Skansi撰写的一本科普计算机科学书籍，属于Undergraduate Topics in Computer Science系列，旨在为初学者提供深度学习的入门知识。这本书阐述了深度学习是人工智能领域的一个重要组成部分，它不仅仅局限于人工神经网络的深层学习，而是逐渐涉足曾经由逻辑AI主导的推理和规划等领域，试图在AI领域占据更广泛的地盘。" 深度学习，作为机器学习的一个子领域，是统计学和人工智能的交叉部分，尤其在人工智能领域中受到广泛关注。传统的浅层神经网络被深度学习所超越，后者通过构建多层非线性变换的神经网络模型，能够学习和理解数据的复杂模式。这种深度结构使得神经网络在图像识别、自然语言处理、语音识别等众多任务上展现出优异的性能。本书可能涵盖了以下几个方面的知识点： 1. **深度学习基础**：介绍神经网络的基本结构，包括输入层、隐藏层和输出层，以及权重和偏置的概念。此外，还会讲解反向传播算法，它是训练神经网络的核心方法。 2. **卷积神经网络（CNN）**：专门用于图像处理的网络结构，通过卷积层提取特征，池化层减少计算量，以及全连接层进行分类。 3. **循环神经网络（RNN）和长短期记忆网络（LSTM）**：适用于序列数据处理，如文本和音频，RNN能捕获时间序列中的依赖关系，而LSTM解决了RNN的梯度消失和爆炸问题。 4. **深度学习优化**：包括损失函数的选择（如均方误差或交叉熵），优化算法（如梯度下降、随机梯度下降、动量法、Adam优化器等），以及学习率调度策略。 5. **模型评估与调优**：涉及验证集和测试集的划分，过拟合和欠拟合的识别，以及正则化技术如Dropout和早停策略。 6. **深度学习框架**：可能会介绍TensorFlow、PyTorch、Keras等流行的深度学习库，以及如何使用它们来构建和训练模型。 7. **强化学习与深度Q网络（DQN）**：将深度学习应用于环境交互的学习问题，DQN允许智能体通过试错来学习最优策略。 8. **生成对抗网络（GAN）**：用于生成新数据，如图像生成，以及在风格转换和数据增强中的应用。 9. **深度学习在其他领域的应用**：例如在自动驾驶、医疗诊断、金融风险评估等方面的应用案例。 10. **未来趋势与挑战**：讨论深度学习的最新发展，如注意力机制、Transformer模型，以及计算效率、可解释性和模型泛化能力等方面的挑战。《Introduction to Deep Learning 2017》会为读者提供一个全面的深度学习入门，引导他们深入理解和实践这一快速发展的领域。通过阅读本书，学生不仅能掌握深度学习的基本原理，还能了解到这一领域的前沿动态和实际应用。

1.1 The Beginnings of Artiﬁcial Neural Networks 3

logical rules. But there was another direction which is characteristic of philosophical

logic: could we model thinking as a human mental process with logical rules? This

is where the neural network history begins, with the seminal paper by Walter Pitts

and Warren McCulloch titled A Logical Calculus of Ideas Immanent in Nervous

Activity and published in the Bulletin of Mathematical Biophysics. A copy of the

paper is available at http://www.cs.cmu.edu/~epxing/Class/10715/

reading/McCulloch.and.Pitts.pdf, and we advise the student to try to

read it to get a sense of how deep learning began.

Warren McCulloch was a philosopher, psychologist and psychiatrist by degree,

but he would work in neurophysiology and cybernetics. He was a vivid character,

embodying many academic stereotypes, and as such was a curious person whose

interests could be described as interdisciplinary. He met the homeless Walter Pitts

in 1942 when he got a job at the Department of Psychiatry at the University of

Chicago, and invited Pitts to come to live with his family. They shared a lifelong

interested in Leibniz, and they wanted to bring his ideas to fruition an create a

machine which could implement logical reasoning.

The two men worked every

night on their idea of capturing reasoning with a logical calculus inspired by the

biological neurons. This meant constructing a formal neuron with capabilities similar

to that of a Turing machine. The paper had only three references, and all of them

are classical works in logic: Carnap’s Logical Syntax of Language [6], Russell’s and

Whitehead’s Principa Mathematica [7] and the Hilbert and Ackermann Grundüge

der Theoretischen Logik. The paper itself approached the problem of neural networks

as a logical one, proceeding from deﬁnitions, over lemmas to theorems.

Their paper introduced the idea of the artiﬁcial neural network, as well as some

of the deﬁnitions we take for granted today. One of these is what would it mean for a

logical predicate to be realizable on a neural network. They divided the neurons in two

groups, the ﬁrst called peripheral afferents (which are now called ‘input neurons’),

and the rest, which are actually output neurons, since at this time there was no hidden

layer—the hidden layer came to play only in the 1970s and 1980s. Neurons can be in

two states, ﬁring and non-ﬁring, and they deﬁne for every neuron i a predicate which

is true when the neuron is ﬁring at the moment t . This predicate is denoted as N

(t).

The solution of a network is then an equivalence of the form N

(t) ≡ B where B is

a conjunction of ﬁrings from the previous moment of the peripheral afferents, and i

is not an input neuron. A sentence like this is realizable in a neural network if and

only if the network can compute it, and all sentences for which there is a network

which computes them are called a temporal propositional expression (TPE). Notice

that TPEs have a logical characterization. The main result of the paper (asides from

deﬁning artiﬁcial neural networks) is that any TPEcan be computed by an artiﬁcial

neural network. This paper would be cited later by John von Neumann as a major

inﬂuence in his own work. This is just a short and incomplete glimpse into this

exciting historical paper, but let us return to the story of the second protagonist.

This was 15 years before artiﬁcial intelligence was deﬁned as a scientiﬁc ﬁeld.

4 1 From Logic to Cognitive Science

Walter Pitts was an interesting person, and, one could argue, the father of artiﬁcial

neural networks. At the age of 12, he ran away from home and hid in a library,

where he read Principia Mathematica [7] by the famous logician Bertrand Russell.

Pitts contacted Russell, who invited him to come to study at Cambridge under his

tutorship, but Pitts was still a child. Several years later, Pitts, now a teenager, found

out that Russell was holding a lecture at the University of Chicago. He met with

Russell in person, and Russell told him to go and meet his old friend from Vienna,

the logician Rudolph Carnap, who was a professor there. Carnap gave Pitts his

seminal book Logical Syntax of Language

[6], which would highly inﬂuence Pitts

in the following years. After his initial contact with Carnap, Pitts disappeared for a

year, and Carnap could not ﬁnd him, but after he did, he used his academic inﬂuence

to get Pitts a student job at the university, so that Pitts does not have to do menial

jobs during days and ghostwrite student papers during nights just to survive.

Another person Pitts met during Russell was Jerome Lettvin, who at the time was

a pre-med student there, and who would later become neurologist and psychiatrist

by degree, but he will also write papers in philosophy and politics. Pitts and Lettvin

became close friends, and would eventually write an inﬂuential paper together (along

with McCulloch and Maturana) titled What the Frog’s Eye Tells the Frog’s Brain in

1959 [8]. Lettvin would also introduce Pitts to the mathematician Norbert Weiner

from MIT who later became known as the father of cybernetics, a ﬁeld colloquially

known as ‘the science of steering’, dedicated to studying system control both in

biological and artiﬁcial systems. Weiner invited Pitts to come to work at MIT (as

a lecturer in formal logic) and the two men worked together for a decade. Neural

networks were at this time considered to be a part of cybernetics, and Pitts and

McCulloch were very active in the ﬁeld, both attending the Macy conferences, with

McCulloch becoming the president of the American Society for Cybernetics in 1967–

1968. During his stay at Chicago, Pitts also met the theoretical physicist Nicolas

Rashevsky, who was a pioneer in mathematical biophysics, a ﬁeld which tried to

explain biological processes with a combination of logic and physics. Physics might

seem distant to neural networks, but in fact, we will soon discuss the role physicists

played in the history of deep learning.

Pitts would remain connected with the University, but he had minor jobs there due

to his lack of formal academic credentials, and in 1944 was hired by the Kellex Cor-

poration (with the help of Weiner), which participated in the Manhattan project. He

detested the authoritarian General Groves (head of the Manhattan project), and would

play pranks to mock the strict and sometimes meaningless rules that he enacted. He

was granted an Associate of Arts degree (2-year degree) by the University of Chicago

as a token of recognition of his 1943 paper, and this would remain the only academic

degree he ever earned. He has never been fond of the usual academic procedures and

this posed a major problem in his formal education. As an illustration, Pitts attended a

The author has a fond memory of this book, but beware: here be dragons. The book is highly

complex due to archaic notation and a system quite different from today’s l ogic, but it is a worthwhile

read if you manage to survive the ﬁrst 20 pages.

1.1 The Beginnings of Artiﬁcial Neural Networks 5

course taught by professor Wilfrid Rall (the pioneer of computational neuroscience),

and Rall remembered Pitts as ‘an oddball who felt compelled to criticize exam ques-

tions rather than answer them’.

In 1952, Norbert Weiner broke all relations with McCulloch, which devastated

Pitts. Weiner wife accused McCulloch that his boys (Pitts and Lettvin) seduced their

daughter, Barbara Weiner. Pitts turned to alcohol to the point that he could not take

care of his dog anymore,

and succumbed to cirrhosis complications in 1969, at the

age of 46. McCulloch died the same year at the age of 70. Both of the Pitts’ papers

we mentioned remain to this day two of the most cited papers in all of science. It is

interesting to note that even though Pitts had direct or mediated contact with most

of the pioneers of AI, Pitts himself never thought about his work as geared towards

building a machine replica of the mind, but rather as a quest to formalize and better

understand human thinking [9], and that puts him squarely in the realm of what is

known today as philosophical logic.

The story of Walter Pitts is a story of inﬂuences of ideas and of collaboration

between scientists of different backgrounds, and in a way a neural network nicely

symbolizes this interaction. One of the main aims of this book is to (re-)introduce

neural networks and deep learning to all the disciplines

which contributed to the

birth and formation of the ﬁeld, but currently shy away from it. The majority of the

story about Walter Pitts we presented is taken from a great article named The man

who tried to redeem the world with logic by Amanda Gefter published in Nautilus

[10] and the paper Walter Pitts by Neil R. Smalheiser [9], both of which we highly

recommend.

1.2 The XOR Problem

In the 1950s, the Dartmouth conference took place and the interest of the newly born

ﬁeld of artiﬁcial intelligence in neural networks was evident from the very conference

manifest. Marvin Minsky, one of the founding fathers of AI and participant to the

Dartmouth conference was completing his dissertation at Princeton in 1954, and the

title was Neural Nets and the Brain Model Problem. Minsky’s thesis addressed several

technical issues, but it became the ﬁrst publication which collected all up to date

results and theorems on neural networks. In 1951, Minsky built a machine (funded

A Newfoundland, name unknown.

An additional point here is the great inﬂuence of Russell and Carnap on Pitts. It i s a great shame

that many logicians today do not know of Pitts, and we hope the present volume will help bring the

story about this amazing man back to the community from which he arose, and that he will receive

the place he deserves.

And any other scientiﬁc discipline which might be interested in studying or using deep neural

networks.

Also, there is a webpage on Pitts http://www.abstractnew.com/2015/01/walter-

pitts-tribute-to-unknown-genius.html worth visiting.

6 1 From Logic to Cognitive Science

by the Air Force Ofﬁce of Scientiﬁc Research) which implemented neural networks

called SNARC (Stochastic Neural Analog Reinforcement Calculator), which was the

ﬁrst major computer implementation of a neural network. As a bit of trivia, Marvin

Minsky was an advisor to Arthur C. Clarke’s and Stanley Kubrick’s 2001: A Space

Odyssey movie. Also, Isaac Asimov claimed that Marvin Minsky was one of two

people he has ever met whose intelligence surpassed his own (the other one being

Carl Sagan). Minsky will return to our story soon, but ﬁrst let us present another hero

of deep learning.

Frank Rosenblatt received his PhD in Psychology at Cornell University in 1956.

Rosenblatt made a crucial contribution to neural networks, by discovering the per-

ceptron learning rule, a rule which governs how to update the weights of neural

networks, which we shall explore in detail in the forthcoming chapters. His percep-

trons were initially developed as a program on an IBM 704 computer at Cornell

Aeronautical Laboratory in 1957, but Rosenblatt would eventually develop the Mark

I Perceptron, a computer built with the sole purpose of implementing neural net-

works with the perceptron rule. But Rosenblatt did more than just implement the

perceptron. His 1962 book Principles of Neurodynamics [11] explored a number of

architectures, and his paper [12] explored the idea of multilayered networks similar

to modern convolutional networks, which he called C-system, which might be seen

as the theoretical birth of deep learning. Rosenblatt died in 1971 on his 43rd birthday

in a boating accident.

There were two major trends underlying the research in the 1960s. The ﬁrst one

was the results that were delivered by programs working on symbolic reasoning, using

deductive logical systems. The two most notable were the Logic Theorist by Herbert

Simon, Cliff Shaw and Allen Newell, and their later program, the General Problem

Solver [13]. Both programs produced working results, something neural networks

did not. Symbolic systems were also appealing since they seemed to provide control

and easy extensibility. The problem was not that neural networks were not giving

any result, just that the results they have been giving (like image classiﬁcation) were

not really considered that intelligent at the time, compared to symbolic systems

that were proving theorems and playing chess—which were the hallmark of human

intelligence. The idea of this intelligence hierarchy was explored by Hans Moravec

in the 1980s [14], who concluded that symbolic thinking is considered a rare and

desirable aspect of intelligence in humans, but it comes rather natural to computers,

which have much more trouble with reproducing ‘low-level’ intelligent behaviour

that many humans seem to exhibit with no trouble, such as recognizing that an animal

in a photo is a dog, and picking up objects.

The second trend was the Cold War. Starting with 1954, the US military wanted to

have a program to automatically translate Russian documents and academic papers.

Even today people consider playing chess or proving theorems as a higher form of intelligence than

for example gossiping, since they point to the rarity of such forms of intelligence. The rarity of an

aspect of intelligence does not directly correlate with its computational properties, since problems

that are computationally easy to describe are easier to solve regardless of the cognitive rarity in

humans (or machines for that matter).

1.2 The XOR Problem 7

Funding was abundant, but many technically inclined researchers underestimated

the linguistic complexities involved in extracting meaning from words. A famous

example was the back and forth translation from English to Russian and back to

English of the phrase ‘the spirit was willing but the ﬂesh was weak’ which produced

the sentence ‘the vodka was good, but the meat was rotten’. In 1964, there were

some concerns about wasting government money in a dead end, so the National

Research Council formed the Automatic Language Processing Advisory Committee

or ALPAC [13]. The ALPAC report from 1966 cut funding to all machine translation

projects, and without the funding, the ﬁeld lingered. This in turn created turmoil in

the whole AI community.

But the ﬁnal stroke which nearly killed off neural networks came in 1969, from

Marvin Minsky and Seymour Papert [15], in their monumental book Perceptrons:

An Introduction to Computational Geometry. Remember that McCulloch and Pitts

proved that a number of logical functions can be computed with a neural network. It

turns out, as Minsky and Papert showed in their book, they missed a simple one, the

equivalence. The computer science and AI community tend to favour looking at this

problem as the XOR function, which is the negation of an equivalence, but it really

does not matter, since the only thing different is how you place the labels.

It turns out that perceptrons, despite the peculiar representations of the data they

process, are only linear classiﬁers. The perceptron learning procedure is remarkable,

since it is guaranteed to converge (terminate), but it did not add a capability of cap-

turing nonlinear regularities to the neural network. The XOR is a nonlinear problem,

but this is not clear at ﬁrst.

To see the problem, imagine

a simple 2D coordinate

system, with only 0 and 1 on both axes. The XOR of 0 and 0 is 0, and write an O at

coordinates (0, 0). The XOR of 0 and 1 is 1, and now write an X at the coordinates

(0,1). Continue with XOR(1, 0) = 1 and XOR(1, 1) = 0. You should have two Xs

and two Os. Now imagine you are the neural network, and you have to ﬁnd out how

to draw a curve to separate the Xs from Os. If you can draw anything, it is easy. But

you are not a modern neural network, but a perceptron, and you must use a straight

line—no curves. It soon becomes obvious that this is impossible.

The problem with

the perceptron was the linearity. The idea of a multilayered perceptron was here, but

it was impossible to build such a device with the perceptron learning rule. And so,

seemingly, no neural network could handle (learn to compute) even the basic logical

operations, something symbolic systems could do in an instant. A quiet darkness fell

across the neural networks, lasting many years. One might wonder what was hap-

pening in the USSR at this time, and the short answer is that cybernetics, as neural

networks were still called in the USSR in this period, was considered a bourgeois

pseudoscience. For a more detailed account, we refer the reader to [16].

The view is further dimmed by the fact that the perceptron could process an image (at least

rudimentary), which intuitively seems to be quite harder than simple logical operations.

Pick up a pen and paper and draw along.

If you wish to try the equivalence instead of XOR, you should do the same but with

EQUIV(0, 0) = 1, EQUIV(0, 1) = 0, EQUIV(1, 0) = 0, EQUIV(1, 1) = 1, keeping the Os for

0 and Xs for 1. You will see it is literally the same thing as XOR in the context of our problem.

剩余195页未读，继续阅读

身份认证购VIP最低享 7 折!

30元优惠券

wcgsir

粉丝: 0

深度学习入门：从逻辑计算到人工智能

深度学习入门：Python实现与理论解析

深度学习入门指南：Ian Goodfellow等人的《DeepLearning》精要

深度学习入门：编纂现有知识与实例讲解

introduction to deep learning

Introduction to Deep Learning with TensorFlow

Introduction to Deep Learning.pdf

Introduction to Deep Learning with Python

Introduction to Deep Learning Using R

Introduction to Deep Learning Using R(Apress,2017)

#GDC2017 NVIDIA GDC Presentations -- Introduction to Deep Learning

最新资源