![](https://csdnimg.cn/release/download_crawler_static/10821460/bg12.jpg)
XX CONTENTS
9.3.2 Simple Reinforcement Learning for Tic-Tac-Toe ............ 380
9.3.3 Role of Deep Learning and a Straw-Man Algorithm ......... 380
9.4 Bootstrapping for Value Function Learning .................. 383
9.4.1 Deep Learning Models as Function Approximators .......... 384
9.4.2 Example: Neural Network for Atari Setting .............. 386
9.4.3 On-Policy Versus Off-Policy Methods: SARSA ............ 387
9.4.4 Modeling States Versus State-Action Pairs ............... 389
9.5 Policy Gradient Methods ............................ 391
9.5.1 Finite Difference Methods ........................ 392
9.5.2 Likelihood Ratio Methods ........................ 393
9.5.3 Combining Supervised Learning with Policy Gradients ........ 395
9.5.4 Actor-Critic Methods .......................... 395
9.5.5 Continuous Action Spaces ........................ 397
9.5.6 Advantages and Disadvantages of Policy Gradients .......... 397
9.6 Monte Carlo Tree Search ............................. 398
9.7 Case Studies ................................... 399
9.7.1 AlphaGo: Championship Level Play at Go ............... 399
9.7.1.1 Alpha Zero: Enhancements to Zero Human Knowledge . . 402
9.7.2 Self-Learning Robots ........................... 404
9.7.2.1 Deep Learning of Locomotion Skills ............. 404
9.7.2.2 Deep Learning of Visuomotor Skills ............. 406
9.7.3 Building Conversational Systems: Deep Learning for Chatbots ... 407
9.7.4 Self-Driving Cars ............................. 410
9.7.5 Inferring Neural Architectures with Reinforcement Learning ..... 412
9.8 Practical Challenges Associated with Safety .................. 413
9.9 Summary ..................................... 414
9.10 Bibliographic Notes ................................ 414
9.10.1 Software Resources and Testbeds .................... 416
9.11 Exercises ..................................... 416
10 Advanced Topics in Deep Learning 419
10.1 Introduction .................................... 419
10.2 Attention Mechanisms .............................. 421
10.2.1 Recurrent Models of Visual Attention ................. 422
10.2.1.1 Application to Image Captioning ............... 424
10.2.2 Attention Mechanisms for Machine Translation ............ 425
10.3 Neural Networks with External Memory .................... 429
10.3.1 A Fantasy Video Game: Sorting by Example ............. 430
10.3.1.1 Implementing Swaps with Memory Operations ....... 431
10.3.2 Neural Turing Machines ......................... 432
10.3.3 Differentiable Neural Computer: A Brief Overview .......... 437
10.4 Generative Adversarial Networks (GANs) ................... 438
10.4.1 Training a Generative Adversarial Network .............. 439
10.4.2 Comparison with Variational Autoencoder ............... 442
10.4.3 Using GANs for Generating Image Data ................ 442
10.4.4 Conditional Generative Adversarial Networks ............. 444
10.5 Competitive Learning .............................. 449
10.5.1 Vector Quantization ........................... 450
10.5.2 Kohonen Self-Organizing Map ..................... 450