没有合适的资源?快使用搜索试试~ 我知道了~
首页cs230讲义-super-cheatsheet-deep-learning
cs230讲义-super-cheatsheet-deep-learning
需积分: 10 9 下载量 56 浏览量
更新于2023-03-16
评论
收藏 5.28MB PDF 举报
对计算机视觉(cv)以及自然语言处理(NLP)两个热门的方向的技术进行总结概述.
资源详情
资源评论
资源推荐
CS 230 – Deep Learning Shervine Amidi & Afshine Amidi
Super VIP Cheatsheet: Deep Learning
Afshine Amidi and Shervine Amidi
November 25, 2018
Contents
1 Convolutional Neural Networks 2
1.1 Overview ................................. 2
1.2 Types of layer .............................. 2
1.3 Filter hyperparameters .......................... 2
1.4 Tuning hyperparameters ......................... 3
1.5 Commonly used activation functions ................... 3
1.6 Object detection ............................. 4
1.6.1 Face verification and recognition ................. 5
1.6.2 Neural style transfer ....................... 5
1.6.3 Architectures using computational tricks ............ 6
2 Recurrent Neural Networks 7
2.1 Overview ................................. 7
2.2 Handling long term dependencies .................... 8
2.3 Learning word representation ...................... 9
2.3.1 Motivation and notations . . . . . . . . . . . . . . . . . . . 9
2.3.2 Word embeddings . . . . . . . . . . . . . . . . . . . . . . . 9
2.4 Comparing words . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5 Language model ............................. 10
2.6 Machine translation ........................... 10
2.7 Attention ................................. 10
3 Deep Learning Tips and Tricks 11
3.1 Data processing ............................. 11
3.2 Training a neural network ........................ 12
3.2.1 Definitions ............................ 12
3.2.2 Finding optimal weights ..................... 12
3.3 Parameter tuning ............................ 12
3.3.1 Weights initialization ...................... 12
3.3.2 Optimizing convergence ..................... 12
3.4 Regularization .............................. 13
3.5 Go od practices .............................. 13
1 Convolutional Neural Networks
1.1 Overview
r Architecture of a traditional CNN –Convolutionalneuralnetworks,alsoknownasCNNs,
are a specific type of neural networks that are generally composed of the following layers:
The convolution layer and the pooling layer can b e fine-tuned with respect to hyperparameters
that are described in the next sections.
1.2 Types of layer
r Convolutional layer (CONV) –Theconvolutionlayer(CONV)usesfiltersthatperform
convolution op erations as it is scanning the input I with respect to its dimensions. Its hyperpa-
rameters include the filter size F and stride S.TheresultingoutputO is called feature map or
activation map.
Remark: the convolution step can be generalized to the 1D and 3D cases as well.
r Pooling (POOL) –Thepoolinglayer(POOL)isadownsamplingoperation,typicallyapplied
after a convolution layer, which does some spatial invariance. In particular, max and average
pooling are special kinds of pooling where the maximum and average value is taken, respectively.
Stanford University 1 Winter 2019
CS 230 – Deep Learning Shervine Amidi & Afshine Amidi
Max pooling Ave r a g e p o o l i n g
Purp ose
Each pooling operation sel e cts the
maximum value of the current view
Each pooling operation averages
the values of the current view
Illustration
Comments
-Preservesdetectedfeatures
-Mostcommonlyused
-Downsamplesfeaturemap
-UsedinLeNet
r Fully Con n e c t e d (FC) –Thefullyconnectedlayer(FC)operatesonaflattenedinputwhere
each input is connected to all neurons. If present, FC layers are usually found towards the end
of CNN architectures and can be used to optimize objectives such as class scores.
1.3 Filter hyperparameters
The convolution layer contains filters for which it is important to know the meaning behind its
hyperparameters.
r Dimensions of a filt er –AfilterofsizeF ◊F applied to an input containing C channels is
a F ◊ F ◊ C volume that performs convolutions on an input of size I ◊ I ◊ C and produces an
output feature map (also called activation map) of size O ◊ O ◊ 1.
Remark: the application of K filters of size F ◊ F results in an output feature map of size
O ◊ O ◊ K.
r Stride –Foraconvolutionalorapoolingoperation,thestrideS denotes the number of pixels
by which the window moves after each operation.
r Zero-padding –Zero-paddingdenotestheprocessofaddingP zeroes to each side of the
boundaries of the input. This value can either be manually specified or automatically set through
one of the three modes detailed below:
Valid Same Full
Value
P =0
P
start
=
Í
SÁ
I
S
Ë≠I+F ≠S
2
Î
P
end
=
Ï
SÁ
I
S
Ë≠I+F ≠S
2
Ì
P
start
œ [[ 0 ,F ≠ 1]]
P
end
= F ≠ 1
Illustration
Purp ose
-Nopadding
-Dropslast
convolution if
dimensions do not
match
-Paddingsuchthatfeature
map size has size
Ï
I
S
Ì
-Outputsizeis
mathematically convenient
-Alsocalled’half’padding
-Maximumpadding
such that end
convolutions are
applied on the limits
of the input
-Filter’sees’theinput
end-to-end
1.4 Tuning hyperpa rameters
r Parameter compatibility in convolution layer –BynotingI the length of the input
volume size, F the length of the filter, P the amount of zero padding, S the stride, then the
output size O of the feature map along that dimension is given by:
O =
I ≠ F + P
start
+ P
end
S
+1
Remark: often times, P
start
= P
end
, P ,inwhichcasewecanreplaceP
start
+ P
end
by 2P in
the formula above.
Stanford University 2 Winter 2019
CS 230 – Deep Learning Shervine Amidi & Afshine Amidi
r Understanding the complexity of the model –Inordertoassessthecomplexityofa
model, it is often useful to determine the number of parameters that its architecture will have.
In a given layer of a convolutional neural network, i t is done as follows:
CONV POOL FC
Illustration
Input size I ◊ I ◊ C I ◊ I ◊ C N
in
Output size O ◊ O ◊ K O ◊ O ◊ C N
out
Number of
parameters
(F ◊ F ◊ C +1)· K 0 (N
in
+1)◊ N
out
Remarks
-Onebiasparameter
per filter
-Inmostcases,S<F
-Acommonchoice
for K is 2C
-Poolingoperation
done channel-wise
-Inmostcases,S = F
-Inputisflattened
-Onebiasparameter
per neuron
-ThenumberofFC
neurons is free of
structural constraints
r Receptive field –Thereceptivefieldatlayerk is the area denoted R
k
◊ R
k
of the input
that each pixel of the k -th activation map can ’see’. By calling F
j
the filter size of layer j and
S
i
the stride value of layer i and with the convention S
0
=1,thereceptivefieldatlayerk can
be computed with the formula:
R
k
=1+
k
ÿ
j=1
(F
j
≠ 1)
j≠1
Ÿ
i=0
S
i
In the example below, we have F
1
= F
2
=3and S
1
= S
2
=1,whichgivesR
2
=1+2· 1+2 · 1=
5.
1.5 Commonly used activation functions
r Rectified Linear Unit –Therectifiedlinearunitlayer(ReLU)isanactivationfunctiong
that is used on all elements of the volume. It aims at introducing non-linearities to the network.
Its variants are summarized in the table below:
ReLU Leaky ReLU ELU
g(z)=max(0,z)
g(z)=max(‘z,z)
with ‘ π 1
g(z)=max(–(e
z
≠ 1),z)
with – π 1
Non-linearity complexities
biologically interpretable
Addresses dying ReLU
issue for negative values
Differentiable everywhere
r Softmax –Thesoftmaxstepcanbeseenasageneralizedlogisticfunctionthattakesasinput
avectorofscoresx œ R
n
and outputs a vector of output probability p œ R
n
through a softmax
function at the end of the architecture. It is defined as follows:
p =
3
p
1
.
.
.
p
n
4
where p
i
=
e
x
i
n
ÿ
j=1
e
x
j
1.6 Object detection
r Types of models –Thereare3maintypesofobjectrecognitionalgorithms,forwhichthe
nature of what is predicted is different. They are described in the table below:
Image classification
Classification
w. localization
Detection
-Classifiesapicture
-Predictsprobability
of object
-Detectsobjectinapicture
-Predictsprobabilityof
object and where it is
located
-Detectsuptoseveralobjects
in a picture
-Predictsprobabilitiesofobjects
and where they are located
Traditional CNN
Simplified YOLO, R-CNN YOLO, R-CNN
r Detection –Inthecontextofobjectdetection,differentmethodsareuseddependingon
whether we just want to locate the object or detect a more complex shape in the image. The
two main ones are summed up in the table below:
Stanford University 3 Winter 2019
剩余12页未读,继续阅读
Duyuankai1992
- 粉丝: 107
- 资源: 71
上传资源 快速赚钱
- 我的内容管理 收起
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
会员权益专享
最新资源
- c++校园超市商品信息管理系统课程设计说明书(含源代码) (2).pdf
- 建筑供配电系统相关课件.pptx
- 企业管理规章制度及管理模式.doc
- vb打开摄像头.doc
- 云计算-可信计算中认证协议改进方案.pdf
- [详细完整版]单片机编程4.ppt
- c语言常用算法.pdf
- c++经典程序代码大全.pdf
- 单片机数字时钟资料.doc
- 11项目管理前沿1.0.pptx
- 基于ssm的“魅力”繁峙宣传网站的设计与实现论文.doc
- 智慧交通综合解决方案.pptx
- 建筑防潮设计-PowerPointPresentati.pptx
- SPC统计过程控制程序.pptx
- SPC统计方法基础知识.pptx
- MW全能培训汽轮机调节保安系统PPT教学课件.pptx
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功
评论0