强化学习：一本经典英文原版教材

需积分: 5 51 浏览量更新于2024-06-28 收藏 86.03MB PDF 举报

"Reinforcement Learning（经典英文原版教材）.pdf 是一本由Richard S. Sutton和Andrew G. Barto合著的机器学习领域的权威教材，主要关注强化学习这一主题。这本书是第二版，首次出版后在机器学习社区广受好评，并被广泛用作教学和研究的参考书。封面设计灵感来源于一个由强化学习系统控制的模拟自行车的轨迹，展示了强化学习在控制任务中的应用。该书遵循Creative Commons Attribution-NonCommercial-NoDerivs 2.0 Generic License的许可协议，允许非商业性的使用和分发，但不得进行修改。本书详细介绍了强化学习的基本概念、算法和理论，旨在帮助读者理解这一领域并进行实践。强化学习是人工智能的一个重要分支，它通过智能体与环境的交互，学习如何采取行动以最大化长期奖励。在第二版中，作者可能会更新和扩展第一版的内容，以反映近年来强化学习在深度学习、游戏策略、机器人控制和其他领域的最新进展。书中的章节可能包括但不限于以下内容： 1. 强化学习的基本原理：介绍强化学习问题的形式化定义，包括状态、动作、奖励和策略等核心概念。 2. 动态规划：讲解动态规划方法，如策略评估和策略改进，用于解决有限状态空间的问题。 3. 值函数近似：讨论当状态空间过大时，如何使用函数近似技术来估计值函数。 4. 蒙特卡洛学习：介绍无模型学习，通过样本平均来估计值函数。 5. 时间差分学习（TD学习）：结合动态规划和蒙特卡洛学习，提出一种更有效的方法来更新值函数估计。 6. 强化学习算法：涵盖Q-learning、SARSA等常用强化学习算法，以及它们的离策略和在线版本。 7. 深度强化学习：介绍如何结合深度学习，构建神经网络来处理高维度输入和复杂决策问题。 8. 稳定性和收敛性：讨论强化学习算法的理论性质，包括学习的稳定性和收敛性分析。 9. 实际应用：展示强化学习在机器人、游戏AI、推荐系统等实际问题中的应用案例。 10. 结论和未来方向：总结强化学习的发展，展望未来的研究趋势和挑战。该书不仅适合计算机科学和人工智能专业的学生，也对研究人员和从业人员极具价值，它提供了深入理解强化学习理论和技术的全面指导。"

xvi Preface to the Second Edition

deserve our deepest gratitude for this edition as well, which wou ld not exist were it not

for their contributions to editi on number one. To that long list we must ad d many others

who contributed speciﬁcally to the second edi ti on . Our students over the many years that

we have taught this material contributed in countless ways: exposi ng errors, o↵ering ﬁxes,

and—not the least—being confused in places where we could have explained things better.

We especially thank Martha Steenstrup for reading and providing detailed comments

throughout. The chapters on psychology and neuroscience could not have been written

without the help of many experts in those ﬁelds. We thank John Moore for his patient

tutoring over many many years on animal learning exper im ents, the ory, and neuroscience,

and for his careful reading of multiple drafts of Chapters 14 and 15. We also thank Matt

Botvinick, Nathaniel Daw, Peter Dayan, and Yael Niv for their penetrating comments on

drafts of these chapter, their essential guidance through the massive literature, and their

interception of many of our errors in early drafts. Of course, the remaining errors in these

chapters—and there must still be some—are totally our own. We t han k Phil Thomas for

helping us make these chapters accessible to non-psychologists and non-neuroscientists,

and we thank Peter Sterling for helping us improve the exposition. We ar e grateful to Jim

Houk for introducing us to the subject of information processing in the basal ganglia and

for alerting us to other relevant aspects of neuroscie nc e. Jos´e Mart´ınez, Terry Sejnowski,

David Silver, Gerry Te s aur o, Georgios Theocharous, and Phil Thomas generously helped

us understand details of their reinforcement learning applications for incl u si on in the

case-studies chapter, and they provided helpful comments on drafts of these sections.

Special thanks are owed to David Silver for helpi ng us better understand Monte Carlo

Tree Search and the DeepMind Go-playing programs. We thank George Konidaris for his

help with the section on the Fourier basis. Emilio Cartoni, Thomas Cederborg, Stefan

Dernbach, Clemens Rosenb aum , Patrick Taylor, Thomas Colin, and Pierre-Luc Bacon

helped us in a number important ways for which we are most grateful .

Sutton wou l d also like to thank the members of the Reinforcement Le arn i ng and

Artiﬁcial Intellige nc e laboratory at the University of Alberta for contributions to the

second edition . He owes a particular debt to Rupam Mahmood for essential contributions

to the treatment of o↵-policy Monte Carlo methods in Chapter 5, to Hamid Maei for

helping develop the perspective on o↵-policy learning presented in Chapter 11, to Eric

Graves for conducting t h e experiments in Chapter 13, to Shangtong Zhang for replicating

and thus verifying almost all the experimental results, to Kr is De Asis for improving

the new technical content of Chapters 7 and 12, and to Harm van Seijen for insights

that led to the separation of n- st ep me t hods from eligibi l ity traces and (along with Hado

van Hasselt) for the ideas involving exact equivalence of forward and backward views of

eligibility traces presented in Chapter 12. Sutton also gratefully acknowledges the support

and freedom he was granted by the Government of Alberta and the National Science and

Engineering Research Council of Canada throughout the period during which t h e second

edition was conceived and written. In particular, he would like to thank Randy Goebel

for creating a supportive and far-sighted environment f or res ear ch in Alberta. He would

also like to thank DeepMind their support in the last six months of writing the book.

Finally, we owe t han k s to t he many careful readers of drafts of t he second edition that

we post e d on the internet. They found many errors that we had missed and alerted us to

potent i al points of confusion.

Preface to the First Edition

We ﬁrst came to focus on what is now known as reinforcement learning in late 1979. We

were both at the University of Massachusetts, working on one of the earliest projects to

revive the idea that networks of neuronlike adaptive elements might prove to be a promising

approach to artiﬁcial adaptive intelligence. The project explored the “heterostatic theor y

of adaptive s ystems” developed by A. Harry Klopf. Harr y’ s work was a rich source of

ideas, and we were permitted to explore them critically and compare them with t h e long

history of prior work in adaptive systems. Our task became one of teasing the ideas apart

and understanding their relationships and relative importance. This continues today,

but in 1979 we came to realize that perhaps the simplest of the ideas, which had long

been taken for granted, had received surprisingly little attention from a computational

perspec t ive. This was simply the idea of a learning system that wants something, that

adapts its behavior in order to maximize a special signal from its environment. This

was the idea of a “hedonistic” learning system, or, as we would say n ow, the idea of

reinforcement learning.

Like others, we had a sense that reinforcement learning had been thoroughly explored

in the early days of cybernetics and artiﬁcial intelligence. On closer inspection , though,

we found that it had been explored only slightly. While rei nf orc em ent learni n g had c l e arl y

motivated some of the earli est computational studies of learning, most of these researchers

had gone on to other things, such as pattern classiﬁcation, supervised learning, and

adaptive control, or they had abandoned the study of learning altogether. As a result, t he

special issues involved in learning how to get something from the environment received

relatively little attention. In retrospec t, focusing on this idea was the critical step that

set this branch of research in motion. Little progress could be made in the computational

study of reinforcement learning until it was recognized that such a fundamental idea had

not yet been thoroughly explored.

The ﬁeld has come a long way since t h en , evolving and maturing in several directions.

Reinforcement learning has gradually become one of the most act i ve research areas in ma-

chine l ear nin g, artiﬁcial intelligence, and neu r al network research. The ﬁeld has developed

strong mathematical foundations and impressive applications. The computational study

of reinforcement learning is now a lar ge ﬁeld, with hundreds of acti ve researchers around

the world in diverse disciplines such as psychology, control theory, artiﬁcial intelligence,

and neuroscience. Particularly important h ave been the contributions establishing and

developing the relationships to the theory of optimal control and dynamic programming.

xvii

xviii Preface to the First Edition

The overall problem of learning from interaction to achieve goals is still far from being

solved, but our understanding of it has improved signiﬁcantly. We can n ow place compo-

nent ideas, such as temporal-di↵erence learning, dynamic programming, and function

approximation, within a coherent perspective wi t h respect to the overall problem.

Our goal in writing this book was to provide a clear an d simple account of the key

ideas and algori t h ms of reinforcement learning. We wanted our treatment to be accessibl e

to readers in all of the related discipli ne s, but we could not cover al l of these pe rs pectives

in detail. For the most part, our treatment t akes the point of view of artiﬁcial intelligence

and engineering. Coverage of connections to other ﬁelds we leave to others or to another

time. We also chose not to produce a rigorous formal treatment of reinf or ce ment learning.

We did not reach for the highest possible level of mathematical abstraction and did not

rely on a theorem–proof format. We tried to choose a level of mathematical detail that

points the mathematically inclined in the right directions without distracting from the

simplicity and potential generality of the underlying ideas.

...

In some sense we have been working toward this book for thirty years, and we have lots

of people t o thank. First, we thank those who have personally helped us develop the overall

view presented in this book: Harry Klopf, for helping us recognize that reinforcement

learning needed to be revived; Chris Watkins, Dimitri Bertsekas, J oh n Tsitsiklis, and

Paul Werbos, for helping us see the value of the relationships to dynamic programming;

John Moore and Jim Kehoe, for insights and inspirations from animal learning theory;

Oliver Selfridge, for emphasizing the breadth and importance of adaptation; and, more

generally, our coll eagu es and stu de nts who have contributed in countless ways: Ron

Williams, Charles Anderson, Satinder Singh, Sridhar Mahadevan, Steve Bradtke, Bob

Crites, Peter Dayan, and Leemon Baird. Our view of reinforcement learning has been

signiﬁcantly enriched by discussions with Paul Cohen, Paul Utgo↵, Martha Steenstrup,

Gerry Tesauro, Mike Jordan, Leslie Kaelbling, Andrew Moore, Chris Atkeson, Tom

Mitchell, Nils Nilsson, Stuart Russell, Tom D ie t t er ich, Tom Dean, and Bob Narendra.

We thank Michael Littman, Gerry Tesauro, Bob Cr i te s, S at in d er Si n gh, an d Wei Zh ang

for provid i ng speciﬁcs of Sections 4.7, 15.1, 15.4, 15.4, and 15.6 respectively. We thank

the Air Force Oﬃce of Scientiﬁc Resear ch, the National Science Foundation, and GTE

Laboratories for their long and farsighted sup port.

We also wish to than k the many people who have read drafts of this book and

provided valuable comment s, including Tom Kalt, John Tsitsiklis, Pawel Cichosz, Olle

G¨allmo, Chuck Anderson, Stuart Russell, Ben Van Roy, Paul Steenstrup , Paul Cohen ,

Sridhar Mahadevan, Jette Ran dlov, Brian Sheppard, Thomas O’Connell, Richard Coggins,

Cristina Versino, John H. Hiett, Andreas Badelt, Jay Ponte, Joe Beck, Justus Piater,

Martha Steenstrup, Satinder Singh, Tommi Jaakkola, Dimitri Bertsekas, Torbj ¨orn Ekman,

Christina Bj¨orkman, Jakob Carlstr¨om, and Olle Palmgren. Finally, we t hank Gwyn

Mitchell for helping in many ways, and Harry Stanton and Bob Prior for being our

champions at MIT Press.

剩余547页未读，继续阅读

承让@

粉丝: 8
资源: 380

强化学习：一本经典英文原版教材

深度强化学习精要：Grokking Deep Reinforcement Learning

2018年强化学习经典教材：《Reinforcement Learning: An Introduction》第二版

强化学习入门经典：Reinforcement Learning_An Introduction

Deep Reinforcement Learning using Cyclical Learning Rates.pdf

Reinforcement Learning-Theory and Algorithm.pdf

Applications of Deep Reinforcement Learning in Communications and Networking.pdf

[2018]Deep Reinforcement Learning for Intelligent transportation predictoin.pdf

Reinforcement Learning An Introduction2019.pdf.zip

Reinforcement Learning：An Introduction.pdf

Distributional Reinforcement Learning with Quantile Regression.pdf

最新资源