DeepLearning（YoshuaBengio2015）

book，Yoshua

需积分: 10 37 浏览量更新于2023-03-16 评论收藏 21.74MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源评论

资源推荐

Deep Learning

Yoshua Bengio

Ian Goodfellow

Aaron Courville

October 03, 2015

CONTENTS

3.6 The Chain Rule of Conditional Probabilities . . . . . . . . . . . . 54

3.7 Independence and Conditional Independence . . . . . . . . . . . 54

3.8 Expectation, Variance and Covariance . . . . . . . . . . . . . . . 55

3.9 Information Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 56

3.10 Common Probability Distributions . . . . . . . . . . . . . . . . . 59

3.11 Useful Properties of Common Functions . . . . . . . . . . . . . . 65

3.12 Bayes’ Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

3.13 Technical Details of Continuous Variables . . . . . . . . . . . . . 67

3.14 Structured Probabilistic Models . . . . . . . . . . . . . . . . . . . 69

3.15 Example: Naive Bayes . . . . . . . . . . . . . . . . . . . . . . . . 70

4 Numerical Computation 77

4.1 Overﬂow and Underﬂow . . . . . . . . . . . . . . . . . . . . . . . 77

4.2 Poor Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4.3 Gradient-Based Optimization . . . . . . . . . . . . . . . . . . . . 79

4.4 Constrained Optimization . . . . . . . . . . . . . . . . . . . . . . 88

4.5 Example: Linear Least Squares . . . . . . . . . . . . . . . . . . . 90

5 Machine Learning Basics 92

5.1 Learning Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 92

5.2 Example: Linear Regression . . . . . . . . . . . . . . . . . . . . . 100

5.3 Generalization, Capacity, Overﬁtting and Underﬁtting . . . . . . 103

5.4 Hyperparameters and Validation Sets . . . . . . . . . . . . . . . . 113

5.5 Estimators, Bias and Variance . . . . . . . . . . . . . . . . . . . . 115

5.6 Maximum Likelihood Estimation . . . . . . . . . . . . . . . . . . 124

5.7 Bayesian Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . 127

5.8 Supervised Learning Algorithms . . . . . . . . . . . . . . . . . . . 134

5.9 Unsupervised Learning Algorithms . . . . . . . . . . . . . . . . . 139

5.10 Weakly Supervised Learning . . . . . . . . . . . . . . . . . . . . . 142

5.11 Building a Machine Learning Algorithm . . . . . . . . . . . . . . 143

5.12 The Curse of Dimensionality and Statistical Limitations of Local

Generalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

II Deep Networks: Modern Practices 156

6 Feedforward Deep Networks 158

6.1 MLPs from the 1980’s . . . . . . . . . . . . . . . . . . . . . . . . 159

6.2 Estimating Conditional Statistics . . . . . . . . . . . . . . . . . . 163

6.3 Parametrizing a Learned Predictor . . . . . . . . . . . . . . . . . 163

6.4 Flow Graphs and Back-Propagation . . . . . . . . . . . . . . . . 175

CONTENTS

6.5 Back-propagation through Random Operations and Graphical

Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

6.6 Universal Approximation Properties and Depth . . . . . . . . . . 192

6.7 Feature / Representation Learning . . . . . . . . . . . . . . . . . 195

6.8 Piecewise Linear Hidden Units . . . . . . . . . . . . . . . . . . . 197

6.9 Historical Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

7 Regularization of Deep or Distributed Models 201

7.1 Regularization from a Bayesian Perspective . . . . . . . . . . . . 203

7.2 Classical Regularization: Parameter Norm Penalty . . . . . . . . 204

7.3 Classical Regularization as Constrained Optimization . . . . . . . 212

7.4 Regularization and Under-Constrained Problems . . . . . . . . . 213

7.5 Dataset Augmentation . . . . . . . . . . . . . . . . . . . . . . . . 214

7.6 Classical Regularization as Noise Robustness . . . . . . . . . . . 216

7.7 Early Stopping as a Form of Regularization . . . . . . . . . . . . 220

7.8 Parameter Tying and Parameter Sharing . . . . . . . . . . . . . . 227

7.9 Sparse Representations . . . . . . . . . . . . . . . . . . . . . . . . 228

7.10 Bagging and Other Ensemble Methods . . . . . . . . . . . . . . . 230

7.11 Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232

7.12 Multi-Task Learning . . . . . . . . . . . . . . . . . . . . . . . . . 235

7.13 Adversarial Training . . . . . . . . . . . . . . . . . . . . . . . . . 236

8 Optimization for Training Deep Models 240

8.1 Optimization for Model Training . . . . . . . . . . . . . . . . . . 241

8.2 Challenges in Neural Network Optimization . . . . . . . . . . . . 246

8.3 Optimization Algorithms I: Basic Algorithms . . . . . . . . . . . 259

8.4 Optimization Algorithms II: Adaptive Learning Rates . . . . . . 265

8.5 Optimization Algorithms III: Approximate Second-Order Methods270

8.6 Optimization Algorithms IV: Natural Gradient

Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280

8.7 Optimization Strategies and Meta-Algorithms . . . . . . . . . . . 282

9 Convolutional Networks 296

9.1 The Convolution Operation . . . . . . . . . . . . . . . . . . . . . 297

9.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300

9.3 Pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306

9.4 Convolution and Pooling as an Inﬁnitely Strong Prior . . . . . . 309

9.5 Variants of the Basic Convolution Function . . . . . . . . . . . . 310

9.6 Structured Outputs . . . . . . . . . . . . . . . . . . . . . . . . . . 316

9.7 Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317

9.8 Eﬃcient Convolution Algorithms . . . . . . . . . . . . . . . . . . 319

iii

CONTENTS

9.9 Random or Unsupervised Features . . . . . . . . . . . . . . . . . 320

9.10 The Neuroscientiﬁc Basis for Convolutional Networks . . . . . . . 321

9.11 Convolutional Networks and the History of Deep Learning . . . . 327

10 Sequence Modeling: Recurrent and Recursive Nets 330

10.1 Unfolding Flow Graphs and Sharing Parameters . . . . . . . . . . 331

10.2 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . 333

10.3 Bidirectional RNNs . . . . . . . . . . . . . . . . . . . . . . . . . . 348

10.4 Encoder-Decoder Sequence-to-Sequence Architectures . . . . . . 348

10.5 Deep Recurrent Networks . . . . . . . . . . . . . . . . . . . . . . 350

10.6 Recursive Neural Networks . . . . . . . . . . . . . . . . . . . . . 352

10.7 The Challenge of Long-Term Dependencies . . . . . . . . . . . . 353

11 Practical methodology 371

11.1 Default Baseline Models . . . . . . . . . . . . . . . . . . . . . . . 373

11.2 Selecting Hyperparameters . . . . . . . . . . . . . . . . . . . . . . 374

11.3 Debugging Strategies . . . . . . . . . . . . . . . . . . . . . . . . . 383

12 Applications 388

12.1 Large Scale Deep Learning . . . . . . . . . . . . . . . . . . . . . . 388

12.2 Computer Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . 396

12.3 Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . 401

12.4 Natural Language Processing and Neural Language Models . . . 405

12.5 Structured Outputs . . . . . . . . . . . . . . . . . . . . . . . . . . 421

12.6 Other Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 423

III Deep Learning Research 432

13 Structured Probabilistic Models for Deep Learning 434

13.1 The Challenge of Unstructured Modeling . . . . . . . . . . . . . . 435

13.2 Using Graphs to Describe Model Structure . . . . . . . . . . . . . 439

13.3 Advantages of Structured Modeling . . . . . . . . . . . . . . . . . 453

13.4 Learning about Dependencies . . . . . . . . . . . . . . . . . . . . 454

13.5 Inference and Approximate Inference over Latent Variables . . . 456

13.6 The Deep Learning Approach to Structured Probabilistic Models 457

14 Monte Carlo Methods 462

14.1 Markov Chain Monte Carlo Methods . . . . . . . . . . . . . . . . 462

14.2 The Diﬃculty of Mixing between Well-Separated Modes . . . . . 464

剩余704页未读，继续阅读

qq_22027245

粉丝: 0
资源: 1

会员权益专享

Deep Learning（Yoshua Bengio 2015）

评论0

会员权益专享

最新资源

Deep Learning（Yoshua Bengio 2015）

评论0

Deep Learning - Yoshua Bengio

中文版Deep Learning - Yoshua Bengio

DeepLearning Yann L., Yoshua B. , Geoffrey H. (2015)

pdf:深度学习 ian goodfellow、yoshua bengio 和 aaron courville pdf

人工智能深度学习参考文献

适合人工智能深度学习的参考文献

deep learning 英文版pdf

计算机毕设英文参考文献

学习基于深度学习的图像特征提取可以参考那些文献或者资料

请给出介绍神经网络发展历史的文献

图像识别有关的文献推荐

图像处理与深度学习相关书籍与作者

推荐几本人工智能最前沿书籍

人工智能十大经典文献

动手学深度学习参考文献的格式

能详细说下训练chatgpt这样的机器人需要看那些书籍吗

中文翻译的深度学习文献

可以推荐一些课程或书籍吗

系统化学习人工智能需要看哪些书籍

会员权益专享

最新资源