没有合适的资源?快使用搜索试试~ 我知道了~
首页an introduction to reinforcement learning by Sutton
资源详情
资源评论
资源推荐
i
Reinforcement Learning:
An Introduction
Second edition, in progress
****Complete Draft****
November 5, 2017
Richard S. Sutton and Andrew G. Barto
c
2014, 2015, 2016, 2017
The text is now complete, except possibly for one more case study to be added to Chapter 16.The
references still need to be thoroughly checked, and an index still needs to b e added. Please send any
errors to rich@richsutton.com and barto@cs.umass.edu. We are also very interested in correcting any
important omissions in the “Bibliographical and Historical Remarks” at the end of each chapter. If
you think of something that really should have been cited, please let us know and we can try to get it
corrected before the final version is printed.
ABradfordBook
The MIT Press
Cambridge, Massachusetts
London, England
ii
In memory of A. Harry Klopf
Contents
Preface to the First Edition ix
Preface to the Second Editi on xi
Summary of Notation xv
1 Introduction 1
1.1 Reinforcement Learning .................................... 1
1.2 Examples ............................................ 4
1.3 Elements of Reinforcement Learning ............................. 5
1.4 Limitations and Scope ..................................... 6
1.5 An Extended Example: Tic-Tac-Toe ............................. 7
1.6 Summary ............................................ 10
1.7 Early History of Reinforcement Learning ........................... 11
I Tabular Solution Methods 18
2 Multi-armed Bandits 19
2.1 A k-armed B and i t Problem .................................. 19
2.2 Action-value Methods ..................................... 20
2.3 The 10-armed Testbed ..................................... 21
2.4 Incremental Implementation .................................. 23
2.5 Tracking a Nonstationary Problem .............................. 25
2.6 Optimistic Initial Values .................................... 26
2.7 Upper-Confidence-Bound Action Selection .......................... 27
2.8 Gradient Bandit Algorithms .................................. 28
2.9 Associative Search (Contextual Bandits) ........................... 31
2.10 Summary ............................................ 32
3 Finite Markov Decision Pr ocesses 37
3.1 The Agent–Environment Interface .............................. 37
3.2 Goals and Rewards ....................................... 42
iii
iv CONTENTS
3.3 Returns and Episodes ..................................... 43
3.4 Unified Notation for Episodic and Continuing Tasks .................... 45
3.5 Policies and Value Functions ................................. 46
3.6 Optimal Policies and Optimal Value Functions ....................... 50
3.7 Optimality and Approximation ................................ 54
3.8 Summary ............................................ 55
4 Dynamic Programming 59
4.1 Policy Evaluation (Prediction) ................................ 60
4.2 Policy Improvement ...................................... 62
4.3 Policy Iteration ......................................... 64
4.4 Value Iteration ......................................... 67
4.5 Asynchronous Dynamic P r ogram mi ng ............................ 69
4.6 Generalized Policy Iteration .................................. 70
4.7 Efficiency of Dynamic Programming ............................. 71
4.8 Summary ............................................ 71
5 Monte Carlo Methods 75
5.1 Monte Carlo Predicti on .................................... 76
5.2 Monte Carlo Estimation of Action Values .......................... 79
5.3 Monte Carlo Control ...................................... 80
5.4 Monte Carlo Control without Exploring Starts ....................... 82
5.5 O↵-policy Prediction via Importance Sampling ....................... 84
5.6 Incremental Implementation .................................. 89
5.7 O↵-policy Monte Carlo Control ................................ 90
5.8 *Discounting-aware Importance Sampling .......................... 92
5.9 *Per-reward Importance Sampling .............................. 93
5.10 Summary ............................................ 94
6 Temporal-Di↵ eren ce Learning 97
6.1 TD Prediction ......................................... 97
6.2 Advantages of TD Prediction Methods ............................ 101
6.3 Optimality of TD(0) ...................................... 103
6.4 Sarsa: On-policy TD Control ................................. 105
6.5 Q-learning: O↵-policy TD Control .............................. 107
6.6 Expected Sarsa ......................................... 109
6.7 Maximization Bias and Double Learni ng ........................... 110
6.8 Games, Afterstates, and Other Special Cases ........................ 112
6.9 Summary ............................................ 113
7 n-step Bootstrapping 115
7.1 n-step TD Prediction ...................................... 115
CONTENTS v
7.2 n-step Sarsa ........................................... 119
7.3 n-step O↵-policy Learning by Importance Sampling .................... 121
7.4 *Per-reward O↵-policy Methods ................................ 122
7.5 O↵-policy Learning Without Importance Sampling:
The n-step Tree Backup Algorithm .............................. 124
7.6 *A Unifying Algorithm: n-step Q() ............................. 126
7.7 Summary ............................................ 129
8 Planning and Learning with Tabular M eth ods 131
8.1 Models and Planni n g ...................................... 131
8.2 Dyna: Integrati n g Planning, Acting, and Learning ..................... 133
8.3 When the Model Is Wrong ................................... 137
8.4 Prioritized Sweeping ...................................... 139
8.5 Expected vs. Sample Updates ................................. 142
8.6 Trajectory Sampling ...................................... 144
8.7 Real-time Dynamic Programming ............................... 146
8.8 Planning at Decision Time ................................... 149
8.9 Heuristic Search ........................................ 150
8.10 Rollout Algorithms ....................................... 152
8.11 Monte Carlo Tree Search .................................... 153
8.12 Summary of the Chapter .................................... 155
8.13 Summary of Part I: Dimensions ................................ 156
II Approximate Solution Me thods 160
9 On-p ol icy Prediction with Approximation 161
9.1 Value-function Approximation ................................. 161
9.2 The Prediction Objective (VE) ................................ 162
9.3 Stochastic-gradient and Semi-gradient Methods ....................... 164
9.4 Linear Methods ......................................... 167
9.5 Feature Construction for Linear Methods .......................... 171
9.5.1 Polynomials ....................................... 172
9.5.2 Fourier Basis ...................................... 173
9.5.3 Coarse Coding ..................................... 175
9.5.4 Tile Coding ....................................... 177
9.5.5 Radial Basis Function s ................................. 181
9.6 Nonlinear Function Appr oximation: Artificial Neural Networks .............. 182
9.7 Least-Squares TD ....................................... 186
9.8 Memory-based Function Ap p roximation ........................... 187
9.9 Kernel-based Function Approximation ............................ 189
9.10 Looking Deeper at On-policy Learning: Interest and Emphasis .............. 190
剩余444页未读,继续阅读
littleredhat31415
- 粉丝: 0
- 资源: 11
上传资源 快速赚钱
- 我的内容管理 收起
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
会员权益专享
最新资源
- RTL8188FU-Linux-v5.7.4.2-36687.20200602.tar(20765).gz
- c++校园超市商品信息管理系统课程设计说明书(含源代码) (2).pdf
- 建筑供配电系统相关课件.pptx
- 企业管理规章制度及管理模式.doc
- vb打开摄像头.doc
- 云计算-可信计算中认证协议改进方案.pdf
- [详细完整版]单片机编程4.ppt
- c语言常用算法.pdf
- c++经典程序代码大全.pdf
- 单片机数字时钟资料.doc
- 11项目管理前沿1.0.pptx
- 基于ssm的“魅力”繁峙宣传网站的设计与实现论文.doc
- 智慧交通综合解决方案.pptx
- 建筑防潮设计-PowerPointPresentati.pptx
- SPC统计过程控制程序.pptx
- SPC统计方法基础知识.pptx
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功
评论0